On 03/22/2016 09:15 AM, Flavio Percoco wrote:
On 21/03/16 21:43 -0400, Adam Young wrote:
I had a good discussion with the Nova folks in IRC today.
My goal was to understand what could talk to what, and the short
according to dansmith
" any node in nova land has to be able to talk to the queue for any
other one for the most part: compute->compute, compute->conductor,
conductor->compute, api->everything. There might be a few exceptions,
but not worth it, IMHO, in the current architecture."
Longer conversation is here:
http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27
Right now, the message queue is a nightmare. All sorts of sensitive
information flows over the message queue: Tokens (including admin)
are the most obvious. Every piece of audit data. All notifications
and all control messages.
Before we continue down the path of "anything can talk to anything"
can we please map out what needs to talk to what, and why? Many of
the use cases seem to be based on something that should be kicked off
by the conductor, such as "migrate, resize, live-migrate" and it
sounds like there are plans to make that happen.
So, let's assume we can get to the point where, if node 1 needs to
talk to node 2, it will do so only via the conductor. With that in
place, we can put an access control rule in place:
I don't think this is going to scale well. Eventually, this will require
evolving the conductor to some sort of message scheduler, which is
pretty much
what the message bus is supposed to do.
I'll limit this to what happens with Rabbit and QPID (AMQP1.0) and leave
0 our of it for now. I'll use rabbit as shorthand for both these, but
the rules are the same for qpid.
For, say, a migrate operation, the call goes to API, controller, and
eventually down to one of the compute nodes. Source? Target? I don't
know the code well enough to say, but let's say it is the source. It
sends an RPC message to the target node. The message goes to the
central broker right now, and then back down to the targen node.
Meanwhile, the source node has set up a reply queue and that queue name
has gone into the message. The target machine responds by getting a
reference to the response queue and sends a message. This message goes
up to the broker, and then down to the the source node.
A man in the middle could sit there and also read off the queue. It
could modify a message, with its own response queue, and happily tranfer
things back and forth.
So, we have the HMAC proposal, which then puts crypto and key
distribution all over the place. Yes, it would guard against a MITM
attack, but the cost in complexity and processor time it high.
Rabbit does not have a very flexible ACL scheme, bascially, a RegEx per
Rabbit user. However, we could easily spin up a new queue for direct
node to node communication that did meet an ACL regex. For example, if
we said that the regex was that the node could only read/write queues
that have its name in them, to make a request and response queue between
node-1 and node-2 we could create a queues
node-1-node-2
node-1-node-2-<uuid>-reply
So, instead of a single queue request, there are two. And conductor
could tell the target node: start listening on this queue.
Or, we could pass the message through the conductor. The request
message goes from node-1 to conductor, where conductor validates the
businees logic of the message, then puts it into the message queue for
node-2. Responses can then go directly back from node-2 to node-1 the
way they do now.
OR...we could set up a direct socket between the two nodes, with the
socket set up info going over the broker. OR we could use a web
server, OR send it over SNMP. Or SMTP, OR TFTP. There are many ways
to get the messages from node to node.
If we are going to use the message broker to do this, we should at
least make it possible to secure it, even if it is not the default approach.
It might be possible to use a broker specific technology to optimize
this, but I am not a Rabbit expert. Maybe there is some way of
filtering messages?
1. Compute nodes can only read from the queue
compute.<name>-novacompute-<index>.localdomain
2. Compute nodes can only write to response queues in the RPC vhost
3. Compute nodes can only write to notification queus in the
notification host.
I know that with AMQP, we should be able to identify the writer of a
message. This means that each compute node should have its own
user. I have identified how to do that for Rabbit and QPid. I
assume for 0mq is would make sense to use ZAP
(http://rfc.zeromq.org/spec:27) but I'd rather the 0mq maintainers
chime in here.
NOTE: Gentle reminder that qpidd has been removed from oslo.messaging.
Yes, but QPID is proton is AMQP1.0 and I did a proof of concept with it
last summer. It supports encryption and authentication over GSSAPI and
is, I think, the best option for securing messaging in an OpenStack
deployment at the moment.
I think you can configure rabbit, amqp1 and other technologies to do
what you're
suggesting here without much trouble. TBH, I'm not sure how many
chances would
be required in Nova (or even oslo.messaging) but I'd dare to say non are
required.
I think it is safe (and sane) to have the same use on the compute
node communicate with Neutron, Nova, and Ceilometer. This will
avoid a false sense of security: if one is compromised, they are all
going to be compromised. Plan accordingly.
Beyond that, we should have message broker users for each of the
components that is a client of the broker.
Applications that run on top of the cloud, and that do not get
presence on the compute nodes, should have their own VHost. I see
Sahara on my Tripleo deploy, but I assume there are others. Either
they completely get their own vhost, or the apps should share one
separate from the RPC/Notification vhosts we currently have. Even
Heat might fall into this category.
Note that those application users can be allowed to read from the
notification queues if necessary. They just should not be using the
same vhost for their own traffic.
Please tell me if/where I am blindingly wrong in my analysis.
I guess my question is: Have you identified things that need to be
changed in
any of the projects for this to be possible? Or is it a pure deployment
recommendation/decision?
There are certainly deployment changes we need to make that help. And we
can likely make it such that the compute nodes can only read from their
own appropriate queues. However, without changing the queue naming
scheme, I can't see how to control who can write to where. Right now,
its a free for all.
I'd argue that any change (assuming changes are required) are likely
to happen
in specific projects (Nova, Neutron, etc) and that once this scenario is
supported, it'll remain a deployment choice to follow it or not. If I
want my
undercloud services to use a single vhost and a single user, I must be
able to
do that. The proposal in this email complicates deployments
significantly,
despite it making sense from a security stand point.
So, nothing I am saying is preventing that. OTOH, there is insufficient
support from the RPC approach to do a more secure ACL.
One more thing. Depending on the messaging technology, having
different virtual
hosts may have an impact on the performance when running under huge
loads given
the fact that the data will be partitioned differently and, therefore,
written/read differently. I don't have good data at hand about this,
sorry.
So, I think that performance can be optimized many ways, including
having multiple Brokers involved in a deployment. I've seen
architecture diagrams to that effect, but have not had to put it in to
production myself.
Flavio
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
