On 06/04/15 22:55, Angus Salkeld wrote:
Hi all
For quite some time we (Heat team) have wanted to be able to send
messages to our
users (by user I do not mean the Operator, but the User that is
interacting with the client).
What do I mean by "user messages", and how do they differ from our
current log messages
and notifications?
- Our current logs are for the operator and have information that the
user should not have
(ip addresses, hostnames, configuration options, other tenant info etc..)
- Our notifications (that Ceilometer uses) *could* be used, but I am not
sure if it quite fits.
(they seem a bit heavy weight for a log message and aimed at higher
level events)
These messages could be (based on Heat's use case):
- Specific user oriented log messages (distinct from our normal operator
logs)
- Deprecation messages (if they are using old resource
properties/template features)
- Progress and resource state changes (an application doesn't want to
poll an api for a state change)
- Automated actions (autoscaling events, time based actions)
- Potentially integrated server logs (from in guest agents)
I wanted to raise this to "[all]" as it would be great to have a general
solution that
all projects can make use of.
What do we have now:
- The user can not get any kind of log message from services. The
closest thing
ATM is the notifications in Ceilometer, but I have the feeling that
these have a different aim.
- nova console log
- Heat has a DB "event" table for users (we have long wanted to get rid
of this)
What do other clouds provide:
- https://devcenter.heroku.com/articles/logging
- https://cloud.google.com/logging/docs/
- https://aws.amazon.com/blogs/aws/cloudwatch-log-service/
- http://aws.amazon.com/cloudtrail/
(other examples...)
What are some options we could investigate:
1. remote syslog
The user provides a rsyslog server IP/port and we send their
messages to that.
[pros] simple, and the user could also send their server's log
messages to the same
rsyslog - great visibility into what is going on.
There are great tools like loggly/logstash/papertrailapp
that source logs from remote syslog
It leaves the user in control of what tools they get to use.
[cons] Would we become a spam agent (just sending traffic to an
IP/Port) - I guess that's how remote syslog
works. I am not sure if this is an issue or not?
This might be a lesser solution for the use case of "an
application doesn't want to poll an api for a state change"
I am not sure how we would integrate this with horizon.
2. Zaqar
We send the messages to a queue in Zaqar.
[pros] multi tenant OpenStack project for messaging!
[cons] I don't think Zaqar is installed in most installations (tho'
please correct me here if this
is wrong). I know Mirantis does not currently support
I think you're correct for now, but I also think that the ability to
send messages to the user side is a valuable enough feature to convince
many cloud operators to deploy it. Everybody wins in that case.
Zaqar, so that would be a problem for me.
There is not the level of external tooling like in option
"1" (logstash and friends)
IMO whatever solution we choose is going to end up requiring the same
semantics as Zaqar: durable storage, timeouts of stale messages,
arbitrary scale-out, multi-tenancy with Keystone auth, pub-sub, and so
on. That leaves us with 3 options:
1) Use Zaqar
2) Write a ground-up replacement for Zaqar. I hope we can agree that
this is insane. Particularly if our reason for not using Zaqar is that
it isn't widely deployed enough yet.
3) Write or make use of something much simpler than Zaqar that
implements only the exact subset of Zaqar's semantics that we need.
However, IMHO that is very likely to turn out to be substantially all of
them, and once again this is unlikely to solve the problem of wide
deployment before Zaqar.
So, in conclusion, +1 Zaqar.
However, there are some other interesting questions posed by your email.
- Do we need a separate service to condition the output to a different
protocol? IMHO no - or, rather, not beyond the ones already proposed for
Zaqar (long polling, WebSockets, SNS-style notifications). Even if there
was some better protocol (in principle I'm not opposed to Atom, for
example), I think we'd benefit more by just adding support for it in
Zaqar - if it works for this use case then it will work for others that
users commonly have.
- What should be feeding inputs into the queue? Some service that
consumes the existing oslo messaging notifications and sanitises them
for the user? Or would every service publish its user notifications
directly to the queue? I think this might vary on a case-by-case basis.
For things like log messages, warnings and the like, I can see that
having a single place to configure it would be valuable. For other
things, like the hooks we just added in Heat to allow the user to insert
their own actions into Heat's workflow, it may make more sense for the
user to explicitly provide the Zaqar topic they want notification on.
Which leads to a final question:
- Are there different use cases here that require different solutions?
I'm pretty sure that Zaqar is the Right Thing for things like Heat
workflow hooks, Ceilometer alarms, triggering Mistral workflows, &c.
However we should at least consider the possibility that there's a
better fit for the more logging-like things. As I mentioned, I think on
the back end they will likely look quite different. I'd like to think
that the user-facing part could be the same, but it would be interesting
to hear from anyone who disagrees. In particular, if we were to go as
far as allowing users to use the same mechanism for their own
application logging, then Zaqar would probably not be the best channel
for it.
cheers,
Zane.
3. Other options:
Please chip in with suggestions/links!
https://blueprints.launchpad.net/heat/+spec/user-visible-logs
Regards
Angus
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
