Hi Frode, On 6/26/25 1:50 PM, Frode Nordahl wrote: > On 25.06.2025 14:58, Dumitru Ceara wrote: >> On 6/23/25 6:23 PM, Frode Nordahl wrote: >>> Hello, All, >>> >> >> Hi Frode, >> >>> Apologies for being late to the discussion, but just wanted to document >>> our interest in this as mentioned in the last IRC meeting and just now >>> in the A/V meeting. >>> >>> >>> On 19.06.2025 16:18, Dumitru Ceara via dev wrote: >>>> Hi Numan, >>>> >>>> On 6/18/25 11:38 PM, Numan Siddique wrote: >>>>> On Wed, Jun 18, 2025 at 5:00 PM Dumitru Ceara <dce...@redhat.com> >>>>> wrote: >>>>>> >>>>>> Hi Han, >>>>>> >>>>>> On 6/18/25 7:40 PM, Han Zhou wrote: >>>>>>> Thanks Numan for the proposal. >>>>>>> >>>>>>> On Wed, Jun 18, 2025 at 4:27 AM Dumitru Ceara <dce...@redhat.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Numan, Ales, >>>>>>>> >>>>>>>> On 6/10/25 4:24 PM, Numan Siddique wrote: >>>>>>>>> On Tue, Jun 10, 2025 at 2:04 AM Ales Musil <amu...@redhat.com> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jun 5, 2025 at 9:44 PM Numan Siddique <num...@ovn.org> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 5, 2025 at 11:33 AM Dumitru Ceara >>>>>>>>>>> <dce...@redhat.com> >>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 6/4/25 6:24 PM, Numan Siddique wrote: >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Numan, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Dumitru and Numan, >>>>>>>>>> >>>>>>>>>> I think the project might be a good idea in general, just adding >>>>>>>>>> my opinion on some parts below. >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> There's a need to configure the provider bridge with specific >>>>>>> OpenFlow >>>>>>>>>>>>> rules after packets leave the OVN pipeline and enter via the >>>>>>>>>>>>> patch >>>>>>>>>>>>> port. >>>>>>>>>>>>> >>>>>>>>>>>>> To simplify this for CMS, I propose utilizing OVN logical >>>>>>>>>>>>> flows. >>>>>>> This >>>>>>>>>>>>> would eliminate the need for CMS to manage direct OpenFlow >>>>>>> connections >>>>>>>>>>>>> and programming. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This seems very interesting! >>>>>>>>>>>> >>>>>>>>>>>>> To achieve this, I've developed a new service within OVN >>>>>>>>>>>>> called >>>>>>>>>>>>> `ovn-pr-controller` (pr = provider). Here's a high-level >>>>>>>>>>>>> overview: >>>>>>>>>>>>> >>>>>>>>>>>>> A new database, `OVN_Provider`, is created with two main >>>>>>>>>>>>> tables: >>>>>>>>>>>>> `PR_Bridge` and `Logical_Flow`. >>>>>>>>>>>>> >>>>>>>>>>>>> `ovn-pr-controller` connects to this database, translates >>>>>>>>>>>>> logical >>>>>>>>>>>>> flows to OpenFlow rules, and programs the bridges. >>>>>>>>>>>>> >>>>>>>>>>>>> CMS adds logical flows for managed provider bridges by >>>>>>>>>>>>> connecting to >>>>>>>>>>>>> the `OVN_Provider` database. >>>>>>>>>>>>> >>>>>>>>>>>>> CMS can define the pipeline as needed. >>>>>>>>>>>>> >>>>>>>>>>>>> An `ovn-prctl` utility (similar to `ovn-nbctl`) is used to >>>>>>>>>>>>> program >>>>>>>>>>>>> logical flows. >>>>>>>>>>>>> >>>>>>>>>>>>> Example Usage: >>>>>>>>>>>>> >>>>>>>>>>>>> # Add a provider bridge >>>>>>>>>>>>> `ovn-prctl add-br br-ext` >>>>>>>>>>>>> >>>>>>>>>>>>> # Add logical flows >>>>>>>>>>>>> `ovn-prctl add-flow br-ext 0 100 "inport == \\"patch-port\\"" >>>>>>>>>>>>> "ct_snat_zone = 1000; next;"` >>>>>>>>>>>>> `ovn-prctl add-flow br-ext 0 0 "1” "next;"` >>>>>>>>>>>>> >>>>>>>>>>>>> `ovn-prctl add-flow br-ext 1 1000 "ip4" "ct_snat;"` >>>>>>>>>>>>> >>>>>>>>>>>>> `ovn-prctl add-flow br-ext 2 1000 "ip4 && ct.new && ct.trk && >>>>>>> ip4.src >>>>>>>>>>>>> == 10.0.0.11" "ct_snat(100.64.0.11); next;"` >>>>>>>>>>>>> >>>>>>>>>>>>> `ovn-prctl add-flow br-ext 2 0 "ip4" "next;"` >>>>>>>>>>>>> >>>>>>>>>>>>> `ovn-prctl add-flow br-ext 3 100 "ip4 && ip4.dst == >>>>>>>>>>>>> 52.92.128.0/17" >>>>>>>>>>>>> "tun.id = 1000; tun_ip4.dst = 10.100.100.1; eth.dst = >>>>>>>>>>>>> 4c:96:14:14:01:b0; outport = \\"vxlan0\\"; output;"` >>>>>>>>>>>>> >>>>>>>>>>>>> `ovn-prctl add-flow br-ext 3 0 “1” “output;”` >>>>>>>>>>>>> >>>>>>>>>>>>> I'd like to get the community's feedback on whether this >>>>>>>>>>>>> service >>>>>>> would >>>>>>>>>>>>> be a valuable addition to OVN. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I wonder if we need these to be separate databases/processes. >>>>>>> Wouldn't >>>>>>>>>>>> it make sense to expose this in the NB database directly? >>>>>>>>>>>> >>>>>>>>>>>> With your proposal, these new ovn-pr-controllers would have to >>>>>>> connect >>>>>>>>>>>> to the central OVN_Provider database from all chassis. But we >>>>>>> already >>>>>>>>>>>> have the ovn-controller connection to the Southbound. Could we >>>>>>> reuse that? >>>>>>>>>>> >>>>>>>>>>> I forgot to mention that the OVN provider ovsdb-server and the >>>>>>>>>>> ovn-pr-controller needs >>>>>>>>>>> to be run on each chassis. >>>>>>>>>>> >>>>>>>> >>>>>>>> I see. However, I think it gives even more flexibility if the OVN >>>>>>>> provider database is centralized. Then the CMS central component >>>>>>>> (e.g., >>>>>>>> neutron for OpenStack) could just write "logical flows" there and >>>>>>>> rely >>>>>>>> on per-chassis ovn-pr-controllers to translate that to OpenFlow. >>>>>>>> >>>>>>> While it brings flexibility, we should also keep in mind that the >>>>>>> central >>>>>>> approach could introduce another choke point for scale. Even if we >>>>>>> reuse SB >>>>>>> DB as the central component, the new ovn-pr-controllers process >>>>>>> would >>>>>>> require new connections to the SB DB. >>>>>>> >>>>>> >>>>>> True, but I was not really thinking of using the SB DB as the central >>>>>> component for ovn-pr-controllers but rather a new database >>>>>> server. And >>>>>> I really hope it wouldn't require storing as many logical flows (and >>>>>> data in general) as we store in the SB today. >>>>>> >>>>> >>>>> Thanks for the comments and the discussion. >>>>> >>>>> Just FYI - I thought we are in favour of having a separate project. >>>>> So I reworked a bit and renamed the project to - ovnbr controller (ovn >>>>> bridge controller) >>>>> - https://github.com/numansiddique/ovnbr/tree/initial_commits >>>>> >>>>> But I can move back the code to ovn project. So not a problem. >>>>> >>>>> >>>>> >>>>>>> Supporting both central and distributed may be an overkill at least >>>>>>> as a >>>>>>> start, if we could decide which one is more practical from a CMS >>>>>>> point of >>>>>>> view. >>>>>>> >>>>>> >>>>>> Maybe, but the "distributed" case would probably mean more work >>>>>> for the >>>>>> CMS on the node side (a per chassis CMS component is definitely >>>>>> required >>>>>> then to provision the per chassis database). >>>>>> >>>>>> I do understand however that we'd need a way to abstract >>>>>> chassis-specific resources (e.g., OVS ports) in the other, >>>>>> "centralized", case. And that might be a significant disadvantage. >>>>>> >>>>>>>> On second thought, if designed properly, there's probably no need >>>>>>>> to add >>>>>>>> a restriction to colocate the new database and ovn-pr- >>>>>>>> controller. We >>>>>>>> could just support both types of deployments: >>>>> >>>>> @Dumitru Ceara If I understand correctly, what you're suggesting >>>>> is that >>>>> CMS can decide to run ovnbr database either central or locally - for >>>>> each chassis. >>>>> >>>> >>>> Yes, that's what I was thinking of. >>> >>> fwiw; We would be interested in the distributed database for the PR >>> controller. >>> >>> Our interest surfaced after a discussion with Jakub Libosvar on how to >>> approach downstream OpenStack Neutron consumption of OVN native BGP >>> support, where he raised the following use case: >>> * Single NIC chassis. >>> * Said NIC is operated by OVS, for simplicity and purpose of discussion, >>> let's say the user space data path. >>> >>> >>> Now let's mix this with OVN native BGP features such as: >>> * Learning the default gateway through route exchange. >>> * Routing IPv4 prefixes over IPv6 next hop (aka. BGP Unnumbered). >>> * BGP protocol redirect to ensure BGP next hop address is the link-local >>> address of a OVN LR on the integration bridge (which may be required for >>> this to work with older versions of FRR on some ToR equipment due to >>> lack of support for 3rd party link-local next hop). >>> >>> >>> With a centralized Southbound database, which a CMS such as OpenStack >>> deploys, we would have a chicken and egg problem, because flows are >>> needed in the integration bridge for the ovn-controller to connect to >>> the Southbound database, and ... you get the picture. >>> >>> I imagine we could make parts of this work better with a distributed >>> database for a OVN Provider Controller. >>> >>> I guess we could even task this distributed controller with "priming" >>> the integration bridge with a simple set of "static flows" enough to get >>> it bootstrapped in such topologies (or could this specific task even be >>> a feature of the regular controller?). >>> >> >> I'm just trying to make sure I understood this last part correctly. By >> "regular controller" do you mean "ovn-controller" or some other (CMS) >> node-specific component? > > I was referring to the "ovn-controller". >
Thanks for the clarification! > The need for a L3 interface on every chassis for BGP generates a lot of > per-chassis configuration, which "pollutes" the Northbound database for > CMSs that uses centralized OVN DBs for the entire deployment. > > The idea of a distributed database for the pr-controller made me think > this distributed database could potentially also be consumed by the ovn- > controller for per-chassis configuration in cases where more advanced > higher layer features are required, and the CMS does not want to re- > implement those in logical flows using the "pr-controller" themselves. > Wouldn't it then make sense to just use ovn-controller for managing all bridges (br-int and also all provider OVS bridges listed in the distributed "Provider Database") in all cases? Thanks, Dumitru > -- > Frode Nordahl > >>>>> I think that would work too. Only disadvantage is that CMS has to >>>>> ensure that the bridge names and physical interface names >>>>> have to be the same. Although we can definitely find a way to handle >>>>> this. >>>>> >>>>> The present proposal schema - >>>>> https://github.com/numansiddique/ovnbr/blob/initial_commits/ovn- >>>>> br.ovsschema#L66 >>>>> has only a Bridge table and no ports defined. In the present proposal >>>>> CMS can add >>>>> logical flows like - "inport == eth1 && ....." and ovnbr-controller >>>>> expects an OVS interface - eth1 in the OVS bridge. >>>>> >>>> >>>> Right, we'd need a way to represent chassis specific information if we >>>> support a central ovnbr database. Probably at least: OVS ports, >>>> conntrack zones, provider bridge names. >>>> >>>>> >>>>> >>>>> >>>>>>>> - ovn-pr-controller & database on the same chassis >>>>>>>> - (multiple) ovn-pr-controller connecting to a database on a >>>>>>>> different >>>>>>>> node (e.g., colocated with the Southbound) >>>>>>>> >>>>>>>>>>> This is basically to allow a CMS agent running on each chassis >>>>>>>>>>> to use >>>>>>>>>>> logical flows >>>>>>>>>>> and program them in the OVN provider database, instead of >>>>>>>>>>> talking >>>>>>>>>>> OpenFlow directly. >>>>>>>>>>> >>>>>>>>>>> OVN has a very nice way of abstracting OpenFlows using logical >>>>>>>>>>> flows >>>>>>>>>>> and my idea is to leverage it. >>>>>>>>>>> >>>>>>>>>>> ovnkube-node, as you know, programs the OpenFlow rules to the >>>>>>>>>>> provider bridge directly by execing >>>>>>>>>>> "ovs-ofctl". The OVN kubernetes community could possibly use >>>>>>>>>>> OVN >>>>>>>>>>> provider controller. >>>>>>>> >>>>>>>> Another thing the CMS node component (ovnkube-node in the example >>>>>>>> above) >>>>>>>> needs to do is reconcile flows on the "provider" bridge on restart. >>>>>>>> >>>>>>>> This new OVN provider controller would also remove the need for >>>>>>>> the CMS >>>>>>>> to do that, which is an advantage. >>>>>>>> >>>>>>> >>>>>>> It is definitely valuable, and no objection from me. But I think >>>>>>> it is >>>>>>> better to ask opinion from CMS component maintainers if the OVS >>>>>>> pipeline >>>>>>> for the provider bridge is complex enough to justify this. Also, >>>>>>> how much >>>>>>> would it help by managing logical flows through an OVSDB versus >>>>>>> managing >>>>>>> open-flow rules though ovs-ofctl. Keep in mind that the abstraction >>>>>>> here is >>>>>>> not at the level of the NB DB which focuses on logical topologies - >>>>>>> here >>>>>>> the abstraction is logical-flows (something at the SB level), so >>>>>>> the CMS >>>>>>> still needs to manage the pipeline rather than just logical >>>>>>> topology. >>>>> >>>>> That's correct. Expectation is that CMS knows what it is doing, >>>>> And we have the requirement to add flows in the provider bridges and >>>>> hence thought >>>>> of this abstraction to >>>>> - leverage the OVN logical flows >>>>> - and to leverage the openflow handling of ovn-controller. >>>>> >>>>> >>>>> My personal preference is also to have it part of OVN so that it is >>>>> easier to manage. >>>>> >>>>> But as Ales mentioned, we have many logical actions like - >>>>> put_dhcp_opts etc (which use controller action) >>>>> which will not work with the ovn-bridge-controller. Probably >>>>> ovn-bridge-controller can maintain its own symbol table. >>>>> >>>>> >>>>> Thanks >>>>> Numan >>>>> >>>>> >>>>>>> >>>>>>> Adding @Tim Rozet <tro...@redhat.com> @Girish Moodalbail >>>>>>> <gmoodalb...@gmail.com> from ovn-k8s to comment. >>>>>>> >>>>>> >>>>>> Adding Jakub Libosvar too, from the neutron side, as we briefly >>>>>> touched >>>>>> on the content of this proposal in a discussion while trying to >>>>>> onboard >>>>>> the OVN BGP support in neutron. IMO a centralized "provider network >>>>>> database" might fit that specific use case quite well because the >>>>>> open >>>>>> flow rules that needed to be added to the provider bridge were quite >>>>>> generic. >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>> I'm not suggesting this, but just giving an example or a >>>>>>>>>>> potential >>>>>>> user. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I believe it could be useful, but I'm unsure if it should be >>>>>>>>>>>>> integrated into OVN directly or be a separate project within >>>>>>>>>>>>> `ovn-org`. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> In my opinion it would be better as a standalone project within >>>>>>>>>> ovn-org. There is couple of reasons why: >>>>>>>>>> >>>>>>>>>> 1) Separation between ovn actions and the new ones. I suppose the >>>>>>>>>> ovn-pr-controller won't support like put_arp etc. in other words >>>>>>>>>> actions that require userspace handling. >>>>>>>>>> >>>>>>>>>> 2) There are certain expectations about parameters, mostly ct >>>>>>>>>> actions >>>>>>>>>> and it feels a little strange to populate a specific register in >>>>>>>>>> order >>>>>>>>>> to get the action working. >>>>>>>>>> >>>>>>>>>> 3) Maintenance cost to some extent, standalone project might be >>>>>>>>>> easier to maintain, it can be written in different language if it >>>>>>>>>> would be better fit. >>>>>>>>>> >>>>>>>>>> 4) The abstraction can be generated automatically to some extent >>>>>>>>>> so most of the code might be actually a way to generate the >>>>>>>>>> parsing >>>>>>>>>> etc. >>>>>>>>>> >>>>>>>>>> I'm fully aware that this would require some duplication. With >>>>>>>>>> that >>>>>>>>>> said, this just my opinion and I won't block any other decision. >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Ales, >>>>>>>>> >>>>>>>>> Thanks for the replies. I agree with your points. I think it >>>>>>>>> makes >>>>>>>>> sense to have >>>>>>>>> a separate project even if the code is duplicated. >>>>>>>>> >>>>>>>> >>>>>>>> I'm not sure I see anything blocking the ovn-pr-controller (or >>>>>>>> whatever >>>>>>>> we decide to call it) code from being part of the ovn-org/ovn >>>>>>>> project. >>>>>>>> >>>>>>>> It can be a standalone daemon (similar to controller/ovn- >>>>>>>> controller or >>>>>>>> controller-vtep/ovn-controller-vtep). >>>>>>>> >>>>>>>> It can even be written in a different language (e.g., rust) if we >>>>>>>> wish. >>>>>>>> >>>>>>>> Keeping it in the same project actually reduces maintenance cost >>>>>>>> because >>>>>>>> we can just reuse the expr parsing library from OVN directly. >>>>>>>> We can >>>>>>>> also reuse all the build/test infra (if we wish). >>>>>>>> >>>>>>>> It would also mean we wouldn't have to come up with different >>>>>>>> release >>>>>>>> strategies, contribution guidelines, etc. >>>>>>>> >>>>>>>> Also, with downstream in mind, it might be (slightly) easier for >>>>>>>> distros >>>>>>>> to package this new binary if its code lived in ovn-org/ovn. >>>>>>>> >>>>>>> >>>>>>> In my option it is really not good to duplicate components in >>>>>>> different >>>>>>> repos. Unless we find ways to expose logical flow translation as a >>>>>>> library >>>>>>> and share between the projects, I am inclined to use OVN repo >>>>>>> directly. >>>>>>> >>>>>>> Thanks, >>>>>>> Han >>>>>>> >>>>>>>>> Implementing in a different language would be nice but I think it >>>>>>>>> will >>>>>>>>> take a lot >>>>>>>>> of effort to implement expr parsing and openflow encoding. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Numan >>>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dumitru >>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I guess it depends a bit on the approach chosen for >>>>>>>>>>>> implementation >>>>>>> but >>>>>>>>>>>> at a first glance this seems as a good candidate for ovn-org >>>>>>>>>>>> to me. >>>>>>>>>>>> >>>>>>>>>>>>> Three possible options are: >>>>>>>>>>>>> >>>>>>>>>>>>> - Integrate the `OVN provider controller` into `ovn-org/ >>>>>>>>>>>>> ovn`. >>>>>>>>>>>>> >>>>>>>>>>>>> - Create a separate project within `ovn-org` (which would >>>>>>>>>>>>> require >>>>>>>>>>>>> duplicating some files like `lib/actions.c`). >>>>>>>>>>>>> >>>>>>>>>>>>> - Do not pursue this. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Listing it here because it's on the "con" side of things. :) I >>>>>>> wonder >>>>>>>>>>>> if we'll end up duplicating all OVS actions into OVN logical >>>>>>>>>>>> flow >>>>>>> actions. >>>>>>>>>>> >>>>>>>>>>> Actually that's my goal. Instead of using OpenFlow rules >>>>>>>>>>> directly, a >>>>>>>>>>> user can express >>>>>>>>>>> their pipeline as logical flows and not worry about the >>>>>>>>>>> intricacies of >>>>>>>>>>> openflow syntax. >>>>>>>>>>> It does add another layer of abstraction for sure. >>> >>> >>> With maintenance cost in mind, we definitively prefer hosting this in >>> the main repository, and I think it fits nicely in there. >>> >>> -- >>> Frode Nordahl >>> >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Numan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> I welcome your thoughts and would like to know if other OVN >>>>>>>>>>>>> users >>>>>>> have >>>>>>>>>>>>> similar requirements. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> A proof-of-concept is available here: >>>>>>>>>>>>> >>>>>>> https://github.com/numansiddique/ovn/tree/ >>>>>>> provider_controller_support. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If there is a consensus in pursuing this further, I'll >>>>>>>>>>>>> work on >>>>>>>>>>>>> refining the patches and submit them as RFC to start with. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> In general I have no objections to an RFC. (CC the rest of the >>>>>>> maintainers). >>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Numan >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Dumitru >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Ales >>>>>>>>> >>>>>>> >>>>>> >> >> Thanks, >> Dumitru >> > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
