On 25.06.2025 14:58, Dumitru Ceara wrote:
On 6/23/25 6:23 PM, Frode Nordahl wrote:
Hello, All,
Hi Frode,
Apologies for being late to the discussion, but just wanted to document
our interest in this as mentioned in the last IRC meeting and just now
in the A/V meeting.
On 19.06.2025 16:18, Dumitru Ceara via dev wrote:
Hi Numan,
On 6/18/25 11:38 PM, Numan Siddique wrote:
On Wed, Jun 18, 2025 at 5:00 PM Dumitru Ceara <dce...@redhat.com> wrote:
Hi Han,
On 6/18/25 7:40 PM, Han Zhou wrote:
Thanks Numan for the proposal.
On Wed, Jun 18, 2025 at 4:27 AM Dumitru Ceara <dce...@redhat.com>
wrote:
Hi Numan, Ales,
On 6/10/25 4:24 PM, Numan Siddique wrote:
On Tue, Jun 10, 2025 at 2:04 AM Ales Musil <amu...@redhat.com>
wrote:
On Thu, Jun 5, 2025 at 9:44 PM Numan Siddique <num...@ovn.org>
wrote:
On Thu, Jun 5, 2025 at 11:33 AM Dumitru Ceara <dce...@redhat.com>
wrote:
On 6/4/25 6:24 PM, Numan Siddique wrote:
Hello,
Hi Numan,
Hi Dumitru and Numan,
I think the project might be a good idea in general, just adding
my opinion on some parts below.
There's a need to configure the provider bridge with specific
OpenFlow
rules after packets leave the OVN pipeline and enter via the
patch
port.
To simplify this for CMS, I propose utilizing OVN logical flows.
This
would eliminate the need for CMS to manage direct OpenFlow
connections
and programming.
This seems very interesting!
To achieve this, I've developed a new service within OVN called
`ovn-pr-controller` (pr = provider). Here's a high-level
overview:
A new database, `OVN_Provider`, is created with two main tables:
`PR_Bridge` and `Logical_Flow`.
`ovn-pr-controller` connects to this database, translates
logical
flows to OpenFlow rules, and programs the bridges.
CMS adds logical flows for managed provider bridges by
connecting to
the `OVN_Provider` database.
CMS can define the pipeline as needed.
An `ovn-prctl` utility (similar to `ovn-nbctl`) is used to
program
logical flows.
Example Usage:
# Add a provider bridge
`ovn-prctl add-br br-ext`
# Add logical flows
`ovn-prctl add-flow br-ext 0 100 "inport == \\"patch-port\\""
"ct_snat_zone = 1000; next;"`
`ovn-prctl add-flow br-ext 0 0 "1” "next;"`
`ovn-prctl add-flow br-ext 1 1000 "ip4" "ct_snat;"`
`ovn-prctl add-flow br-ext 2 1000 "ip4 && ct.new && ct.trk &&
ip4.src
== 10.0.0.11" "ct_snat(100.64.0.11); next;"`
`ovn-prctl add-flow br-ext 2 0 "ip4" "next;"`
`ovn-prctl add-flow br-ext 3 100 "ip4 && ip4.dst ==
52.92.128.0/17"
"tun.id = 1000; tun_ip4.dst = 10.100.100.1; eth.dst =
4c:96:14:14:01:b0; outport = \\"vxlan0\\"; output;"`
`ovn-prctl add-flow br-ext 3 0 “1” “output;”`
I'd like to get the community's feedback on whether this service
would
be a valuable addition to OVN.
I wonder if we need these to be separate databases/processes.
Wouldn't
it make sense to expose this in the NB database directly?
With your proposal, these new ovn-pr-controllers would have to
connect
to the central OVN_Provider database from all chassis. But we
already
have the ovn-controller connection to the Southbound. Could we
reuse that?
I forgot to mention that the OVN provider ovsdb-server and the
ovn-pr-controller needs
to be run on each chassis.
I see. However, I think it gives even more flexibility if the OVN
provider database is centralized. Then the CMS central component
(e.g.,
neutron for OpenStack) could just write "logical flows" there and
rely
on per-chassis ovn-pr-controllers to translate that to OpenFlow.
While it brings flexibility, we should also keep in mind that the
central
approach could introduce another choke point for scale. Even if we
reuse SB
DB as the central component, the new ovn-pr-controllers process would
require new connections to the SB DB.
True, but I was not really thinking of using the SB DB as the central
component for ovn-pr-controllers but rather a new database server. And
I really hope it wouldn't require storing as many logical flows (and
data in general) as we store in the SB today.
Thanks for the comments and the discussion.
Just FYI - I thought we are in favour of having a separate project.
So I reworked a bit and renamed the project to - ovnbr controller (ovn
bridge controller)
- https://github.com/numansiddique/ovnbr/tree/initial_commits
But I can move back the code to ovn project. So not a problem.
Supporting both central and distributed may be an overkill at least
as a
start, if we could decide which one is more practical from a CMS
point of
view.
Maybe, but the "distributed" case would probably mean more work for the
CMS on the node side (a per chassis CMS component is definitely
required
then to provision the per chassis database).
I do understand however that we'd need a way to abstract
chassis-specific resources (e.g., OVS ports) in the other,
"centralized", case. And that might be a significant disadvantage.
On second thought, if designed properly, there's probably no need
to add
a restriction to colocate the new database and ovn-pr-controller. We
could just support both types of deployments:
@Dumitru Ceara If I understand correctly, what you're suggesting
is that
CMS can decide to run ovnbr database either central or locally - for
each chassis.
Yes, that's what I was thinking of.
fwiw; We would be interested in the distributed database for the PR
controller.
Our interest surfaced after a discussion with Jakub Libosvar on how to
approach downstream OpenStack Neutron consumption of OVN native BGP
support, where he raised the following use case:
* Single NIC chassis.
* Said NIC is operated by OVS, for simplicity and purpose of discussion,
let's say the user space data path.
Now let's mix this with OVN native BGP features such as:
* Learning the default gateway through route exchange.
* Routing IPv4 prefixes over IPv6 next hop (aka. BGP Unnumbered).
* BGP protocol redirect to ensure BGP next hop address is the link-local
address of a OVN LR on the integration bridge (which may be required for
this to work with older versions of FRR on some ToR equipment due to
lack of support for 3rd party link-local next hop).
With a centralized Southbound database, which a CMS such as OpenStack
deploys, we would have a chicken and egg problem, because flows are
needed in the integration bridge for the ovn-controller to connect to
the Southbound database, and ... you get the picture.
I imagine we could make parts of this work better with a distributed
database for a OVN Provider Controller.
I guess we could even task this distributed controller with "priming"
the integration bridge with a simple set of "static flows" enough to get
it bootstrapped in such topologies (or could this specific task even be
a feature of the regular controller?).
I'm just trying to make sure I understood this last part correctly. By
"regular controller" do you mean "ovn-controller" or some other (CMS)
node-specific component?
I was referring to the "ovn-controller".
The need for a L3 interface on every chassis for BGP generates a lot of per-chassis
configuration, which "pollutes" the Northbound database for CMSs that uses
centralized OVN DBs for the entire deployment.
The idea of a distributed database for the pr-controller made me think this distributed
database could potentially also be consumed by the ovn-controller for per-chassis
configuration in cases where more advanced higher layer features are required, and the
CMS does not want to re-implement those in logical flows using the
"pr-controller" themselves.
--
Frode Nordahl
I think that would work too. Only disadvantage is that CMS has to
ensure that the bridge names and physical interface names
have to be the same. Although we can definitely find a way to handle
this.
The present proposal schema -
https://github.com/numansiddique/ovnbr/blob/initial_commits/ovn-
br.ovsschema#L66
has only a Bridge table and no ports defined. In the present proposal
CMS can add
logical flows like - "inport == eth1 && ....." and ovnbr-controller
expects an OVS interface - eth1 in the OVS bridge.
Right, we'd need a way to represent chassis specific information if we
support a central ovnbr database. Probably at least: OVS ports,
conntrack zones, provider bridge names.
- ovn-pr-controller & database on the same chassis
- (multiple) ovn-pr-controller connecting to a database on a
different
node (e.g., colocated with the Southbound)
This is basically to allow a CMS agent running on each chassis
to use
logical flows
and program them in the OVN provider database, instead of talking
OpenFlow directly.
OVN has a very nice way of abstracting OpenFlows using logical
flows
and my idea is to leverage it.
ovnkube-node, as you know, programs the OpenFlow rules to the
provider bridge directly by execing
"ovs-ofctl". The OVN kubernetes community could possibly use OVN
provider controller.
Another thing the CMS node component (ovnkube-node in the example
above)
needs to do is reconcile flows on the "provider" bridge on restart.
This new OVN provider controller would also remove the need for
the CMS
to do that, which is an advantage.
It is definitely valuable, and no objection from me. But I think it is
better to ask opinion from CMS component maintainers if the OVS
pipeline
for the provider bridge is complex enough to justify this. Also,
how much
would it help by managing logical flows through an OVSDB versus
managing
open-flow rules though ovs-ofctl. Keep in mind that the abstraction
here is
not at the level of the NB DB which focuses on logical topologies -
here
the abstraction is logical-flows (something at the SB level), so
the CMS
still needs to manage the pipeline rather than just logical topology.
That's correct. Expectation is that CMS knows what it is doing,
And we have the requirement to add flows in the provider bridges and
hence thought
of this abstraction to
- leverage the OVN logical flows
- and to leverage the openflow handling of ovn-controller.
My personal preference is also to have it part of OVN so that it is
easier to manage.
But as Ales mentioned, we have many logical actions like -
put_dhcp_opts etc (which use controller action)
which will not work with the ovn-bridge-controller. Probably
ovn-bridge-controller can maintain its own symbol table.
Thanks
Numan
Adding @Tim Rozet <tro...@redhat.com> @Girish Moodalbail
<gmoodalb...@gmail.com> from ovn-k8s to comment.
Adding Jakub Libosvar too, from the neutron side, as we briefly touched
on the content of this proposal in a discussion while trying to onboard
the OVN BGP support in neutron. IMO a centralized "provider network
database" might fit that specific use case quite well because the open
flow rules that needed to be added to the provider bridge were quite
generic.
I'm not suggesting this, but just giving an example or a potential
user.
I believe it could be useful, but I'm unsure if it should be
integrated into OVN directly or be a separate project within
`ovn-org`.
In my opinion it would be better as a standalone project within
ovn-org. There is couple of reasons why:
1) Separation between ovn actions and the new ones. I suppose the
ovn-pr-controller won't support like put_arp etc. in other words
actions that require userspace handling.
2) There are certain expectations about parameters, mostly ct
actions
and it feels a little strange to populate a specific register in
order
to get the action working.
3) Maintenance cost to some extent, standalone project might be
easier to maintain, it can be written in different language if it
would be better fit.
4) The abstraction can be generated automatically to some extent
so most of the code might be actually a way to generate the parsing
etc.
I'm fully aware that this would require some duplication. With that
said, this just my opinion and I won't block any other decision.
Hi Ales,
Thanks for the replies. I agree with your points. I think it makes
sense to have
a separate project even if the code is duplicated.
I'm not sure I see anything blocking the ovn-pr-controller (or
whatever
we decide to call it) code from being part of the ovn-org/ovn
project.
It can be a standalone daemon (similar to controller/ovn-
controller or
controller-vtep/ovn-controller-vtep).
It can even be written in a different language (e.g., rust) if we
wish.
Keeping it in the same project actually reduces maintenance cost
because
we can just reuse the expr parsing library from OVN directly. We can
also reuse all the build/test infra (if we wish).
It would also mean we wouldn't have to come up with different release
strategies, contribution guidelines, etc.
Also, with downstream in mind, it might be (slightly) easier for
distros
to package this new binary if its code lived in ovn-org/ovn.
In my option it is really not good to duplicate components in
different
repos. Unless we find ways to expose logical flow translation as a
library
and share between the projects, I am inclined to use OVN repo
directly.
Thanks,
Han
Implementing in a different language would be nice but I think it
will
take a lot
of effort to implement expr parsing and openflow encoding.
Thanks
Numan
Thanks,
Dumitru
I guess it depends a bit on the approach chosen for
implementation
but
at a first glance this seems as a good candidate for ovn-org
to me.
Three possible options are:
- Integrate the `OVN provider controller` into `ovn-org/ovn`.
- Create a separate project within `ovn-org` (which would
require
duplicating some files like `lib/actions.c`).
- Do not pursue this.
Listing it here because it's on the "con" side of things. :) I
wonder
if we'll end up duplicating all OVS actions into OVN logical flow
actions.
Actually that's my goal. Instead of using OpenFlow rules
directly, a
user can express
their pipeline as logical flows and not worry about the
intricacies of
openflow syntax.
It does add another layer of abstraction for sure.
With maintenance cost in mind, we definitively prefer hosting this in
the main repository, and I think it fits nicely in there.
--
Frode Nordahl
Thanks
Numan
I welcome your thoughts and would like to know if other OVN
users
have
similar requirements.
A proof-of-concept is available here:
https://github.com/numansiddique/ovn/tree/provider_controller_support.
If there is a consensus in pursuing this further, I'll work on
refining the patches and submit them as RFC to start with.
In general I have no objections to an RFC. (CC the rest of the
maintainers).
Thanks,
Numan
Regards,
Dumitru
Thanks,
Ales
Thanks,
Dumitru
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
