Re: [openstack-dev] [neutron] How could an L2 agent extension access agent methods ?

[Rossella Sblendido]

Hi Ihar,

wow, good job!!
Sorry for the very slow reply.
I really like your proposal...some comments inline.

On 12/03/2015 04:46 PM, Ihar Hrachyshka wrote:
Hi,

Small update on the RFE. It was approved for Mitaka, assuming we come up
with proper details upfront thru neutron-specs process.

In the meantime, we have found more use cases for flow management among
features in development: QoS DSCP, also the new OF based firewall
driver. Both authors for those new features independently realized that
agent does not currently play nice with flows set by external code due
to its graceful restart behaviour when rules with unknown cookies are
cleaned up. [The agent uses a random session uuid() to mark rules that
belong to its current run.]

Before I proceed, full disclosure: I know almost nothing about OpenFlow
capabilities, so some pieces below may make no sense. I tried to come up
with high level model first and then try to map it to available OF
features. Please don’t hesitate to comment, I like to learn new stuff! ;)

I am not an expert either so I encourage people to chime in here.


I am thinking lately on the use cases we collected so far. One common
need for all features that were seen to be interested in proper
integration with Open vSwitch agent is to be able to manage feature
specific flows on br-int and br-tun. There are other things that
projects may need, like patch ports, though I am still struggling with
the question of whether it may be postponed or avoided for phase 1.

There are several specific operation 'kinds' that we should cover for
the RFE:
- managing flows that modify frames in-place;
- managing flows that redirect frames.

There are some things that should be considered to make features
cooperate with the agent and other extensions:
- feature flows should have proper priorities based on their ‘kind’
(f.e. in-place modification probably go before redirections);
- feature flows should survive flow reset that may be triggered by the
agent;
- feature flows should survive flow reset without data plane disruption
(=they should support graceful restart:
https://review.openstack.org/#/c/182920).

With that in mind, I see the following high level design for the flow
tables:

- table 0 serves as a dispatcher for specific features;
- each feature gets one or more tables, one per flow ‘kind’ needed;
- for each feature table, a new flow entry is added to table 0 that
would redirect to feature specific table; the rule will be triggered
only if OF metadata is not updated inside the feature table (see the
next bullet); the rule will have priority that is defined for the ‘kind’
of the operation that is implemented by the table it redirects to;
-  each feature table will have default actions that will 1) mark OF
metadata for the frame as processed by the feature; 2) redirect back to
table 0;
- all feature specific flow rules (except dispatcher rules) belong to
feature tables;

Now, the workflow for extensions that are interested in setting flows
would be:
- on initialize() call, extension defines feature tables it will need;

Do you mean this in a dynamic way or every extension will have tables 
assigned, basically hard-coded? I prefer the second way so we have more 
controls of the tables that are currently used.

it passes the name of the feature table and the ‘kind’ of the actions it
will execute; with that, the following is initialized by the agent: 1)

It would be nice to pass also a filter to match some packets. We 
probably don't want to send all the packet to the feature table, the 
extension can define that.

table 0 dispatcher entry to redirect frames into feature table; the
entry has the priority according to the ‘kind’ of the table; 2) the

I think we need to define the priority better. According to what you 
wrote we assign priority based on "in-place modification probably go 
before redirections" not sure if it's enough. What happens if we have 
two features that both requires in place-modifications? How do we 
prioritize them? Are we going to allow 2 extension at the same time? Let 
me think more about this...It would be nice to have some real world 
example...

actual feature table with two default rules (update metadata and push
back to table 0);
- whenever extension needs to add a new flow rule, it passes the
following into the agent: 1) table name; 2) flow specific parameters
(actions, priority, ...)

Since the agent will manage setting flows for extensions, it will be
able to use the active agent cookie for all feature flows; next time the
agent is restarted, it should be able to respin extension flows with no
data plane disruption. [Note: we should make sure that on agent restart,
we call to extensions *before* we clean up stale flow rules.]

I like this :)

That design will hopefully allow us to abstract interaction with flows
from extensions into management code inside the agent. It should
guarantee extensions cooperate properly assuming they properly define
their priorities thru ‘kinds’ of tables they have.

It is also assumed that existing flow based features integrated into the
agent (dvr? anti-spoofing?) will eventually move to the new flow table
management model.

Good candidates for a POC of this model?

thanks,

Rossella

I understand that the model does not reflect how do feature processing
for existing OF based features in the agent. It may require some smart
workarounds to allow non-disruptive migration to new flow table setup.

It would be great to see the design bashed hard before I start to put it
into spec format. Especially if it’s not sane. :)

Ihar

Mathieu Rohon <mathieu.ro...@gmail.com> wrote:

Thanks ihar!

On Thu, Nov 19, 2015 at 2:32 PM, Ihar Hrachyshka <ihrac...@redhat.com>
wrote:
UPD: now that we have some understanding what’s needed from l2 agent
extension mechanism to cater for interested subprojects (and now that
we see that probably the agent in focus right now is OVS only), we may
move to RFE step. I reported the following RFE for the feature:

https://bugs.launchpad.net/neutron/+bug/1517903

It may require BP if drivers team will request one.

Cheers,

Ihar

Ihar Hrachyshka <ihrac...@redhat.com> wrote:

Reviving the thread.

On the design summit session dedicated to agent and plugin extensions
[1] the following was stated for l2 agent extensions (I appreciate if
someone checks me on the following though):

- current l2 agent extensions mechanism lacks insight into agent
details like bridges or vlan maps;

- in some cases, we don’t care about extension portability across
multiple agents, so it’s not of concern if some of them use
implementation details like bridges to set specific flows, or to wire
up some additional ports to them;

- that said, we still don’t want extensions to have unlimited access
to agent details; the rationale for hard constraints on what is seen
inside extensions is that we cannot support backwards compatibility
for *all* possible internal attributes of an agent; instead, we should
explicitly define where we can make an effort to provide stable API
into agent details, and what’s, on contrary, beyond real life use
cases and hence can be left to be broken/refactored as neutron
developers see fit; this API can be agent specific though;

- agent details that are to be passed into extensions should be driven
by actual use cases. There were several subprojects mentioned in the
session that are assumed to lack enough access to agent attributes to
do their job without patching core ovs agent files. Those are:
BGP-VPN, SFC, (anything else?) Those subprojects that are interested
in extending l2 agent extension framework are expected to come up with
a list of things missing in current implementation, so that neutron
developers can agree on proper abstractions to provide missing details
to extensions. For that goal, I set up a new etherpad to collect
feedback from subprojects [2].

Once we collect use cases there and agree on agent API for extensions
(even if per agent type), we will implement it and define as stable
API, then pass objects that implement the API into extensions thru
extension manager. If extensions support multiple agent types, they
can still distinguish between which API to use based on agent type
string passed into extension manager.

I really hope we start to collect use cases early so that we have time
to polish agent API and make it part of l2 extensions earlier in
Mitaka cycle.

[1]: https://etherpad.openstack.org/p/mitaka-neutron-core-extensibility
[2]: https://etherpad.openstack.org/p/l2-agent-extensions-api-expansion

Ihar

Ihar Hrachyshka <ihrac...@redhat.com> wrote:

On 30 Sep 2015, at 12:53, Miguel Angel Ajo <mangel...@redhat.com> wrote:



Ihar Hrachyshka wrote:
On 30 Sep 2015, at 12:08, thomas.mo...@orange.com wrote:

Hi Ihar,

Ihar Hrachyshka :
Miguel Angel Ajo :
Do you have a rough idea of what operations you may need to do?
Right now, what bagpipe driver for networking-bgpvpn needs to interact
with is:
- int_br OVSBridge (read-only)
- tun_br OVSBridge (add patch port, add flows)
- patch_int_ofport port number (read-only)
- local_vlan_map dict (read-only)
- setup_entry_for_arp_reply method (called to add static ARP entries)
Sounds very tightly coupled to OVS agent.
Please bear in mind, the extension interface will be available from
different agent types
(OVS, SR-IOV, [eventually LB]), so this interface you're talking about
could also serve as
a translation driver for the agents (where the translation is
possible), I totally understand
that most extensions are specific agent bound, and we must be able to
identify
the agent we're serving back exactly.
Yes, I do have this in mind, but what we've identified for now seems
to be OVS specific.
Indeed it does. Maybe you can try to define the needed pieces in high
level actions, not internal objects you need to access to. Like ‘-
connect endpoint X to Y’, ‘determine segmentation id for a network’ etc.
I've been thinking about this, but would tend to reach the conclusion
that the things we need to interact with are pretty hard to abstract
out into something that would be generic across different agents.
Everything we need to do in our case relates to how the agents use
bridges and represent networks internally: linuxbridge has one bridge
per Network, while OVS has a limited number of bridges playing
different roles for all networks with internal segmentation.

To look at the two things you  mention:
- "connect endpoint X to Y" : what we need to do is redirect the
traffic destinated to the gateway of a Neutron network, to the thing
that will do the MPLS forwarding for the right BGP VPN context (called
VRF), in our case br-mpls (that could be done with an OVS table too) ;
that action might be abstracted out to hide the details specific to
OVS, but I'm not sure on how to  name the destination in a way that
would be agnostic to these details, and this is not really relevant to
do until we have a relevant context in which the linuxbridge would
pass packets to something doing MPLS forwarding (OVS is currently the
only option we support for MPLS forwarding, and it does not really
make sense to mix linuxbridge for Neutron L2/L3 and OVS for MPLS)
- "determine segmentation id for a network": this is something really
OVS-agent-specific, the linuxbridge agent uses multiple linux bridges,
and does not rely on internal segmentation

Completely abstracting out packet forwarding pipelines in OVS and
linuxbridge agents would possibly allow defining an interface that
agent extension could use without to know about anything specific to
OVS or the linuxbridge, but I believe this is a very significant taks
to tackle.

If you look for a clean way to integrate with reference agents, then
it’s something that we should try to achieve. I agree it’s not an easy
thing.

Just an idea: can we have a resource for traffic forwarding, similar
to security groups? I know folks are not ok with extending security
groups API due to compatibility reasons, so maybe fwaas is the place
to experiment with it.

Hopefully it will be acceptable to create an interface, even it
exposes a set of methods specific to the linuxbridge agent and a set
of methods specific to the OVS agent.  That would mean that the agent
extension that can work in both contexts (not our case yet) would
check the agent type before using the first set or the second set.

The assumption of the whole idea of l2 agent extensions is that they
are agent agnostic. In case of QoS, we implemented a common QoS
extension that can be plugged in any agent [1], and a set of backend
drivers (atm it’s just sr-iov [2] and ovs [3]) that are selected based
on the driver type argument passed into the extension manager [4][5].
Your extension could use similar approach to select the backend.

[1]:
https://git.openstack.org/cgit/openstack/neutron/tree/neutron/agent/l2/extensions/qos.py#n169

[2]:
https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/mech_sriov/agent/extension_drivers/qos_driver.py

[3]:
https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py

[4]:
https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#n395

[5]:
https://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py#n155


I disagree on the agent-agnostic thing. QoS extension for SR-IOV is
totally not agnostic for OVS or LB, in the QoS case, it's just
accidental that OVS & LB share common bridges now due to the OVS
Hybrid implementation that leverages linux bridge
and iptables.

Wait. The QoS extension has nothing agent backend specific. All it
does is it receives rpc updates for tracked resources and pass them
into qos drivers. Those latter are the bits that implement backend
specific operations. So I am not sure why you say the extension itself
is agent specific: any other amqp based agent in the wild can adopt
the extension as-is, only providing a new backend to load.

I agree on having a well defined interface, on which API is available
to talking back to each agent, and it has to be common, where
it's possible to be common.

It doesn't have to be easy, but it's the way if we want a world where
those commonalities and reusability of extensions can
exist and not be just accidental, but it's not realistic in my opinion
to AIM for it on every shot. I believe we should try where we can
but we should be open to agent specific extensions. The idea of the
extensions is that you can extend specific agents without
being forced to have the main loop hijacked, or eventually having off
tree code plugged into our agents.

Partially, yes. The culprit here is how much the extension API should
know about an agent. We can probably make the extension API completely
extendable by allowing agents to pass any random kwargs into the
extension manager that will forward them into extensions. Note that it
breaks current API for extensions and technically breaks it (not that
I know of any external extensions that could be affected so far).

There we should add support to identify the type of agent the
extension works with (compatibility, versioning, etc..)

We already pass the type into extension manager, and that’s how we
plug in the proper backend driver in QoS.

Does this approach make sense ?

-Thomas

_________________________________________________________________________________________________________________________


Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les
messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere,
deforme ou falsifie. Merci.

This message and its attachments may contain confidential or
privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and
delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have
been modified, changed or falsified.
Thank you.

Note that you should really avoid putting that ^^ kind of signature
into your emails intended for public mailing lists. If it’s
confidential, why do you send it to everyone? And sorry, folks will
copy it without authorisation, for archiving and indexing reasons and
whatnot.

Ihar
__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev