Hi,
Thanks for the input and comments. This is really great.
I would like to propose the following staged development (enable to
stabilize, test, and then optimize):
1. Stage 1 - have the agent detect a change, initially by polling. When
the agent detects and update then it will contact the plugin for a
detailed update about the network.
2. Stage 2 - Event driven support. One option is to have the operating
system notify the agent (as suggested by Darragh) another is to have the
VIF driver notify the agent. I am in favour of the latter. The VIF
driver is essentially creating the new tap device or deleting the
existing tap device. It seems logical that this would drive the update
on the agent.
What I would like to do is a quick POC of the above and then write a
detailed design of the flow so that we can all review. If it "compiles
and runs" on paper then it will speed up the development, testing and
deployment. It will also enable us to document for future reference.
This will also save time with review and the ping/pong with the -1's.
Have a good weekend and thanks for the inputs and comments. Hopefully
next week I'll have an update on the progress.
Thanks
Gary
On 05/11/2012 12:30 AM, Maru Newby wrote:
Thanks Darragh! That should cover kvm. And apparently it's possible
to be notified of vif changes from xen/xcp too, a more xen-savvy
co-worker is tracking down details.
Gary, it sounds like it will be possible to have the agent notified
directly of device changes. What are you thoughts as to modifying
your proposal to take this into account?
Cheers,
Maru
On 2012-05-10, at 2:06 PM, Darragh OReilly wrote:
maybe udev events rules/actions could be installed for add/remove tap
device events
http://www.reactivated.net/writing_udev_rules.html#external-run
------------------------------------------------------------------------
*From:* Maru Newby <mne...@internap.com <mailto:mne...@internap.com>>
*To:* gkot...@redhat.com <mailto:gkot...@redhat.com>
*Cc:* Christopher Wright <chr...@redhat.com
<mailto:chr...@redhat.com>>; netstack@lists.launchpad.net
<mailto:netstack@lists.launchpad.net>
*Sent:* Thursday, 10 May 2012, 18:38
*Subject:* Re: [Netstack] Scalable
Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)
Hi Gary,
I appreciate the effort you've put into condensing the options.
I agree with your suggestion that option 1 is a good starting
point. How will the agent discover changes to tap devices? Can
an agent register for events from linux/kvm or xen, or would the
agent just poll? For all I know agents may do this already, so I
apologize if this is a silly question.
Regarding option 2, I still see no reason to have the vif driver
talk to the agent directly. Ensuring a single point of contact
between quantum clients (of which the vif driver is one) and
quantum, namely the rest interface, limits complexity and will be
easier to maintain and test. If and when performance or other
concerns require direct vif driver to agent communication, we can
go down that road, but as of now it's answering a question that
hasn't been asked. YAGNI.
I would also argue that even RPC communication between the plugin
and agent is gold-plating. The problem at hand is that database
polling doesn't scale well. The simple answer is for the plugin
and agent to communicate directly rather than through a database
intermediary. Adding RPC to the mix is an implementation detail,
pure and simple, and is not cost-free. RPC introduces queue
dependency that can be problematic to debug and as we've seen in
nova can cause performance issues all its own.
I'm all for leaving us open going forward to introduce an RPC
dependency, but I think the most important thing is to create a
clean communication interface between plugin and agent. The
initial implementation can be something simple (and relatively
dependency free) like secured http. The semantics for
implementing and debugging http communication are well-known to
all of us. If and when RPC becomes necessary, it will be
straightforward to plug in a new transport driver.
Let's keep it simple - distributed computing is complicated enough!
Cheers,
Maru
On 2012-05-10, at 8:22 AM, Gary Kotton wrote:
Hi,
Below is a table that lists a number of options, a short
description, their advantages and disadvantages. Hopefully this
can give an idea of the scope and complexity.
*Option*
*Description*
*Advantages*
*Disadvantages*
1 .Agent driving data retrieval from plugin
The agent maintains a list of tap devices. If there is a new
tap device then the agent will request the network information
for this tap device from the quantum plugin. In the case of the
open source ovs and lb (linuxbridge) plugins this is tap + 11
letters of the attachment id.
The agent will send an RCP update about the delta to the plugin.
The plugin will answer accordingly. For example if one or more
tap devices are detected then these are sent to the plugin. For
each new tap device the plugin will sent the network information
(tags etc) and set the database attachment as up. For deletion
they will be removed (or set as down).
Simple
Self contained in Quantum
If there is more than 1 attachment ID with the same prefix of
11 characters then this will not work (this currently is a bug)
The agent will still have to poll the network interfaces.
2 .VIF driver driving retrieval from plugin
The VIF driver updates the plugin about a change, which inturn
updates the relevant agent (this was described in the link
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
)
Event driven.
No polling
VIF driver and agents will need to share communication channels
3. Plugin broadcasting
When the plugin receives a change it broadcasts the change to
all of the registered agents
Relatively simple
Lots of unnecessary messages to agents that do not need to deal
with the traffic
I think that option #1 is a good start. This can later be
optimized to option #2.
Thanks
Gary
On 05/10/2012 10:05 AM, Gary Kotton wrote:
On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
Hi Gary,
Thanks for initiating this. A couple of comments/questions -
1. Do we really need the VIF driver to communicate the agent's
identity;
I am referring to the agent ID being sent by the VIF driver in
the
message? In general, I am not sure if there is a need to have
the VIF
driver send messages/notifications in the first place, but I
perhaps
it's being included as a capability in the framework?
At the moment the open source plugins are not aware of the
agents. The agents poll the data base for updates. The agent ID
enables a agent to regsiter with the plugin, this in trrun
enables the plugin to send a update to the specific agent. The
update is initiated by the VIF driver. In my opinion this does
the following:
1. updates the agents as soon as possible regarding a network
change
2. limits traffic on the network
3. removes the database interface from the agents
2. One model I was thinking of (which is kind of inline with the
existing agent implementations), is where the agents are
smart, and they
know what to do in response to changes in the state of the
logical
Quantum resources. In such cases, the Quantum plugin need not
have to
keep track of sending a message to a particular agent.
Instead, can we
have broadcast messages from the plugin to all the agents? If
the plugin
has to unicast messages to specific agents, then it needs to
maintain a
lot more state/topology information which should not be
mandated for
this sole reason.
I too thought about this option. In a sense the above proposal
is an optimization of what you mention. This comes at the cost
of complexity. The broadcast option is nice when the number of
agents is small. When this is large, then for each network
update there will be NUMBER_OF_AGENT messages sent for each
update. The advantage of what you mention is that the code is
self contained in Quantum.
It may be better to start with the broadcast and then deal with
the optimizations afterwards.
Thanks
Gary
Thanks,
~Sumit.
-----Original Message-----
From: netstack-bounces+snaiksat=cisco....@lists.launchpad.net
<mailto:netstack-bounces+snaiksat=cisco....@lists.launchpad.net>
[mailto:netstack-bounces+snaiksat=cisco....@lists.launchpad.net]
On
Behalf Of Gary Kotton
Sent: Wednesday, May 09, 2012 4:27 AM
To:<netstack@lists.launchpad.net>
<mailto:netstack@lists.launchpad.net>
Subject: [Netstack] Scalable
Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
comms)
Hi,
I have added a very high level description on how to address the
issue.
This can be seen at:
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
ZKgOlpvg/edit
Comments will be greatly appreciated.
Questions:
1. Do we want agents to be backward compatible (that is, still
maintain
the polling code)
2. The generation of the Agent ID
3. Any other ideas or thoughts about the matter?
I'd like to go ahead with a POC and implement this.
Thanks
Gary
--
Mailing list: https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
Post to : netstack@lists.launchpad.net
<mailto:netstack@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
More help : https://help.launchpad.net/ListHelp
--
Mailing list: https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
Post to : netstack@lists.launchpad.net
<mailto:netstack@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
More help : https://help.launchpad.net/ListHelp
--
Mailing list: https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
Post to : netstack@lists.launchpad.net
<mailto:netstack@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
More help : https://help.launchpad.net/ListHelp
--
Mailing list: https://launchpad.net/~netstack
Post to : netstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp
