On 6/23/25 6:23 PM, Frode Nordahl wrote: > Hello, All, > Hi Frode,
> Apologies for being late to the discussion, but just wanted to document > our interest in this as mentioned in the last IRC meeting and just now > in the A/V meeting. > > > On 19.06.2025 16:18, Dumitru Ceara via dev wrote: >> Hi Numan, >> >> On 6/18/25 11:38 PM, Numan Siddique wrote: >>> On Wed, Jun 18, 2025 at 5:00 PM Dumitru Ceara <dce...@redhat.com> wrote: >>>> >>>> Hi Han, >>>> >>>> On 6/18/25 7:40 PM, Han Zhou wrote: >>>>> Thanks Numan for the proposal. >>>>> >>>>> On Wed, Jun 18, 2025 at 4:27 AM Dumitru Ceara <dce...@redhat.com> >>>>> wrote: >>>>>> >>>>>> Hi Numan, Ales, >>>>>> >>>>>> On 6/10/25 4:24 PM, Numan Siddique wrote: >>>>>>> On Tue, Jun 10, 2025 at 2:04 AM Ales Musil <amu...@redhat.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 5, 2025 at 9:44 PM Numan Siddique <num...@ovn.org> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> On Thu, Jun 5, 2025 at 11:33 AM Dumitru Ceara <dce...@redhat.com> >>>>> wrote: >>>>>>>>>> >>>>>>>>>> On 6/4/25 6:24 PM, Numan Siddique wrote: >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Numan, >>>>>>>> >>>>>>>> >>>>>>>> Hi Dumitru and Numan, >>>>>>>> >>>>>>>> I think the project might be a good idea in general, just adding >>>>>>>> my opinion on some parts below. >>>>>>>> >>>>>>>>>> >>>>>>>>>>> There's a need to configure the provider bridge with specific >>>>> OpenFlow >>>>>>>>>>> rules after packets leave the OVN pipeline and enter via the >>>>>>>>>>> patch >>>>>>>>>>> port. >>>>>>>>>>> >>>>>>>>>>> To simplify this for CMS, I propose utilizing OVN logical flows. >>>>> This >>>>>>>>>>> would eliminate the need for CMS to manage direct OpenFlow >>>>> connections >>>>>>>>>>> and programming. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This seems very interesting! >>>>>>>>>> >>>>>>>>>>> To achieve this, I've developed a new service within OVN called >>>>>>>>>>> `ovn-pr-controller` (pr = provider). Here's a high-level >>>>>>>>>>> overview: >>>>>>>>>>> >>>>>>>>>>> A new database, `OVN_Provider`, is created with two main tables: >>>>>>>>>>> `PR_Bridge` and `Logical_Flow`. >>>>>>>>>>> >>>>>>>>>>> `ovn-pr-controller` connects to this database, translates >>>>>>>>>>> logical >>>>>>>>>>> flows to OpenFlow rules, and programs the bridges. >>>>>>>>>>> >>>>>>>>>>> CMS adds logical flows for managed provider bridges by >>>>>>>>>>> connecting to >>>>>>>>>>> the `OVN_Provider` database. >>>>>>>>>>> >>>>>>>>>>> CMS can define the pipeline as needed. >>>>>>>>>>> >>>>>>>>>>> An `ovn-prctl` utility (similar to `ovn-nbctl`) is used to >>>>>>>>>>> program >>>>>>>>>>> logical flows. >>>>>>>>>>> >>>>>>>>>>> Example Usage: >>>>>>>>>>> >>>>>>>>>>> # Add a provider bridge >>>>>>>>>>> `ovn-prctl add-br br-ext` >>>>>>>>>>> >>>>>>>>>>> # Add logical flows >>>>>>>>>>> `ovn-prctl add-flow br-ext 0 100 "inport == \\"patch-port\\"" >>>>>>>>>>> "ct_snat_zone = 1000; next;"` >>>>>>>>>>> `ovn-prctl add-flow br-ext 0 0 "1” "next;"` >>>>>>>>>>> >>>>>>>>>>> `ovn-prctl add-flow br-ext 1 1000 "ip4" "ct_snat;"` >>>>>>>>>>> >>>>>>>>>>> `ovn-prctl add-flow br-ext 2 1000 "ip4 && ct.new && ct.trk && >>>>> ip4.src >>>>>>>>>>> == 10.0.0.11" "ct_snat(100.64.0.11); next;"` >>>>>>>>>>> >>>>>>>>>>> `ovn-prctl add-flow br-ext 2 0 "ip4" "next;"` >>>>>>>>>>> >>>>>>>>>>> `ovn-prctl add-flow br-ext 3 100 "ip4 && ip4.dst == >>>>>>>>>>> 52.92.128.0/17" >>>>>>>>>>> "tun.id = 1000; tun_ip4.dst = 10.100.100.1; eth.dst = >>>>>>>>>>> 4c:96:14:14:01:b0; outport = \\"vxlan0\\"; output;"` >>>>>>>>>>> >>>>>>>>>>> `ovn-prctl add-flow br-ext 3 0 “1” “output;”` >>>>>>>>>>> >>>>>>>>>>> I'd like to get the community's feedback on whether this service >>>>> would >>>>>>>>>>> be a valuable addition to OVN. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I wonder if we need these to be separate databases/processes. >>>>> Wouldn't >>>>>>>>>> it make sense to expose this in the NB database directly? >>>>>>>>>> >>>>>>>>>> With your proposal, these new ovn-pr-controllers would have to >>>>> connect >>>>>>>>>> to the central OVN_Provider database from all chassis. But we >>>>> already >>>>>>>>>> have the ovn-controller connection to the Southbound. Could we >>>>> reuse that? >>>>>>>>> >>>>>>>>> I forgot to mention that the OVN provider ovsdb-server and the >>>>>>>>> ovn-pr-controller needs >>>>>>>>> to be run on each chassis. >>>>>>>>> >>>>>> >>>>>> I see. However, I think it gives even more flexibility if the OVN >>>>>> provider database is centralized. Then the CMS central component >>>>>> (e.g., >>>>>> neutron for OpenStack) could just write "logical flows" there and >>>>>> rely >>>>>> on per-chassis ovn-pr-controllers to translate that to OpenFlow. >>>>>> >>>>> While it brings flexibility, we should also keep in mind that the >>>>> central >>>>> approach could introduce another choke point for scale. Even if we >>>>> reuse SB >>>>> DB as the central component, the new ovn-pr-controllers process would >>>>> require new connections to the SB DB. >>>>> >>>> >>>> True, but I was not really thinking of using the SB DB as the central >>>> component for ovn-pr-controllers but rather a new database server. And >>>> I really hope it wouldn't require storing as many logical flows (and >>>> data in general) as we store in the SB today. >>>> >>> >>> Thanks for the comments and the discussion. >>> >>> Just FYI - I thought we are in favour of having a separate project. >>> So I reworked a bit and renamed the project to - ovnbr controller (ovn >>> bridge controller) >>> - https://github.com/numansiddique/ovnbr/tree/initial_commits >>> >>> But I can move back the code to ovn project. So not a problem. >>> >>> >>> >>>>> Supporting both central and distributed may be an overkill at least >>>>> as a >>>>> start, if we could decide which one is more practical from a CMS >>>>> point of >>>>> view. >>>>> >>>> >>>> Maybe, but the "distributed" case would probably mean more work for the >>>> CMS on the node side (a per chassis CMS component is definitely >>>> required >>>> then to provision the per chassis database). >>>> >>>> I do understand however that we'd need a way to abstract >>>> chassis-specific resources (e.g., OVS ports) in the other, >>>> "centralized", case. And that might be a significant disadvantage. >>>> >>>>>> On second thought, if designed properly, there's probably no need >>>>>> to add >>>>>> a restriction to colocate the new database and ovn-pr-controller. We >>>>>> could just support both types of deployments: >>> >>> @Dumitru Ceara If I understand correctly, what you're suggesting >>> is that >>> CMS can decide to run ovnbr database either central or locally - for >>> each chassis. >>> >> >> Yes, that's what I was thinking of. > > fwiw; We would be interested in the distributed database for the PR > controller. > > Our interest surfaced after a discussion with Jakub Libosvar on how to > approach downstream OpenStack Neutron consumption of OVN native BGP > support, where he raised the following use case: > * Single NIC chassis. > * Said NIC is operated by OVS, for simplicity and purpose of discussion, > let's say the user space data path. > > > Now let's mix this with OVN native BGP features such as: > * Learning the default gateway through route exchange. > * Routing IPv4 prefixes over IPv6 next hop (aka. BGP Unnumbered). > * BGP protocol redirect to ensure BGP next hop address is the link-local > address of a OVN LR on the integration bridge (which may be required for > this to work with older versions of FRR on some ToR equipment due to > lack of support for 3rd party link-local next hop). > > > With a centralized Southbound database, which a CMS such as OpenStack > deploys, we would have a chicken and egg problem, because flows are > needed in the integration bridge for the ovn-controller to connect to > the Southbound database, and ... you get the picture. > > I imagine we could make parts of this work better with a distributed > database for a OVN Provider Controller. > > I guess we could even task this distributed controller with "priming" > the integration bridge with a simple set of "static flows" enough to get > it bootstrapped in such topologies (or could this specific task even be > a feature of the regular controller?). > I'm just trying to make sure I understood this last part correctly. By "regular controller" do you mean "ovn-controller" or some other (CMS) node-specific component? >>> I think that would work too. Only disadvantage is that CMS has to >>> ensure that the bridge names and physical interface names >>> have to be the same. Although we can definitely find a way to handle >>> this. >>> >>> The present proposal schema - >>> https://github.com/numansiddique/ovnbr/blob/initial_commits/ovn- >>> br.ovsschema#L66 >>> has only a Bridge table and no ports defined. In the present proposal >>> CMS can add >>> logical flows like - "inport == eth1 && ....." and ovnbr-controller >>> expects an OVS interface - eth1 in the OVS bridge. >>> >> >> Right, we'd need a way to represent chassis specific information if we >> support a central ovnbr database. Probably at least: OVS ports, >> conntrack zones, provider bridge names. >> >>> >>> >>> >>>>>> - ovn-pr-controller & database on the same chassis >>>>>> - (multiple) ovn-pr-controller connecting to a database on a >>>>>> different >>>>>> node (e.g., colocated with the Southbound) >>>>>> >>>>>>>>> This is basically to allow a CMS agent running on each chassis >>>>>>>>> to use >>>>>>>>> logical flows >>>>>>>>> and program them in the OVN provider database, instead of talking >>>>>>>>> OpenFlow directly. >>>>>>>>> >>>>>>>>> OVN has a very nice way of abstracting OpenFlows using logical >>>>>>>>> flows >>>>>>>>> and my idea is to leverage it. >>>>>>>>> >>>>>>>>> ovnkube-node, as you know, programs the OpenFlow rules to the >>>>>>>>> provider bridge directly by execing >>>>>>>>> "ovs-ofctl". The OVN kubernetes community could possibly use OVN >>>>>>>>> provider controller. >>>>>> >>>>>> Another thing the CMS node component (ovnkube-node in the example >>>>>> above) >>>>>> needs to do is reconcile flows on the "provider" bridge on restart. >>>>>> >>>>>> This new OVN provider controller would also remove the need for >>>>>> the CMS >>>>>> to do that, which is an advantage. >>>>>> >>>>> >>>>> It is definitely valuable, and no objection from me. But I think it is >>>>> better to ask opinion from CMS component maintainers if the OVS >>>>> pipeline >>>>> for the provider bridge is complex enough to justify this. Also, >>>>> how much >>>>> would it help by managing logical flows through an OVSDB versus >>>>> managing >>>>> open-flow rules though ovs-ofctl. Keep in mind that the abstraction >>>>> here is >>>>> not at the level of the NB DB which focuses on logical topologies - >>>>> here >>>>> the abstraction is logical-flows (something at the SB level), so >>>>> the CMS >>>>> still needs to manage the pipeline rather than just logical topology. >>> >>> That's correct. Expectation is that CMS knows what it is doing, >>> And we have the requirement to add flows in the provider bridges and >>> hence thought >>> of this abstraction to >>> - leverage the OVN logical flows >>> - and to leverage the openflow handling of ovn-controller. >>> >>> >>> My personal preference is also to have it part of OVN so that it is >>> easier to manage. >>> >>> But as Ales mentioned, we have many logical actions like - >>> put_dhcp_opts etc (which use controller action) >>> which will not work with the ovn-bridge-controller. Probably >>> ovn-bridge-controller can maintain its own symbol table. >>> >>> >>> Thanks >>> Numan >>> >>> >>>>> >>>>> Adding @Tim Rozet <tro...@redhat.com> @Girish Moodalbail >>>>> <gmoodalb...@gmail.com> from ovn-k8s to comment. >>>>> >>>> >>>> Adding Jakub Libosvar too, from the neutron side, as we briefly touched >>>> on the content of this proposal in a discussion while trying to onboard >>>> the OVN BGP support in neutron. IMO a centralized "provider network >>>> database" might fit that specific use case quite well because the open >>>> flow rules that needed to be added to the provider bridge were quite >>>> generic. >>>> >>> >>> >>> >>> >>>>>>>>> I'm not suggesting this, but just giving an example or a potential >>>>> user. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I believe it could be useful, but I'm unsure if it should be >>>>>>>>>>> integrated into OVN directly or be a separate project within >>>>>>>>>>> `ovn-org`. >>>>>>>> >>>>>>>> >>>>>>>> In my opinion it would be better as a standalone project within >>>>>>>> ovn-org. There is couple of reasons why: >>>>>>>> >>>>>>>> 1) Separation between ovn actions and the new ones. I suppose the >>>>>>>> ovn-pr-controller won't support like put_arp etc. in other words >>>>>>>> actions that require userspace handling. >>>>>>>> >>>>>>>> 2) There are certain expectations about parameters, mostly ct >>>>>>>> actions >>>>>>>> and it feels a little strange to populate a specific register in >>>>>>>> order >>>>>>>> to get the action working. >>>>>>>> >>>>>>>> 3) Maintenance cost to some extent, standalone project might be >>>>>>>> easier to maintain, it can be written in different language if it >>>>>>>> would be better fit. >>>>>>>> >>>>>>>> 4) The abstraction can be generated automatically to some extent >>>>>>>> so most of the code might be actually a way to generate the parsing >>>>>>>> etc. >>>>>>>> >>>>>>>> I'm fully aware that this would require some duplication. With that >>>>>>>> said, this just my opinion and I won't block any other decision. >>>>>>> >>>>>>> >>>>>>> Hi Ales, >>>>>>> >>>>>>> Thanks for the replies. I agree with your points. I think it makes >>>>>>> sense to have >>>>>>> a separate project even if the code is duplicated. >>>>>>> >>>>>> >>>>>> I'm not sure I see anything blocking the ovn-pr-controller (or >>>>>> whatever >>>>>> we decide to call it) code from being part of the ovn-org/ovn >>>>>> project. >>>>>> >>>>>> It can be a standalone daemon (similar to controller/ovn- >>>>>> controller or >>>>>> controller-vtep/ovn-controller-vtep). >>>>>> >>>>>> It can even be written in a different language (e.g., rust) if we >>>>>> wish. >>>>>> >>>>>> Keeping it in the same project actually reduces maintenance cost >>>>>> because >>>>>> we can just reuse the expr parsing library from OVN directly. We can >>>>>> also reuse all the build/test infra (if we wish). >>>>>> >>>>>> It would also mean we wouldn't have to come up with different release >>>>>> strategies, contribution guidelines, etc. >>>>>> >>>>>> Also, with downstream in mind, it might be (slightly) easier for >>>>>> distros >>>>>> to package this new binary if its code lived in ovn-org/ovn. >>>>>> >>>>> >>>>> In my option it is really not good to duplicate components in >>>>> different >>>>> repos. Unless we find ways to expose logical flow translation as a >>>>> library >>>>> and share between the projects, I am inclined to use OVN repo >>>>> directly. >>>>> >>>>> Thanks, >>>>> Han >>>>> >>>>>>> Implementing in a different language would be nice but I think it >>>>>>> will >>>>>>> take a lot >>>>>>> of effort to implement expr parsing and openflow encoding. >>>>>>> >>>>>>> Thanks >>>>>>> Numan >>>>>>> >>>>>> >>>>>> Thanks, >>>>>> Dumitru >>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I guess it depends a bit on the approach chosen for >>>>>>>>>> implementation >>>>> but >>>>>>>>>> at a first glance this seems as a good candidate for ovn-org >>>>>>>>>> to me. >>>>>>>>>> >>>>>>>>>>> Three possible options are: >>>>>>>>>>> >>>>>>>>>>> - Integrate the `OVN provider controller` into `ovn-org/ovn`. >>>>>>>>>>> >>>>>>>>>>> - Create a separate project within `ovn-org` (which would >>>>>>>>>>> require >>>>>>>>>>> duplicating some files like `lib/actions.c`). >>>>>>>>>>> >>>>>>>>>>> - Do not pursue this. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Listing it here because it's on the "con" side of things. :) I >>>>> wonder >>>>>>>>>> if we'll end up duplicating all OVS actions into OVN logical flow >>>>> actions. >>>>>>>>> >>>>>>>>> Actually that's my goal. Instead of using OpenFlow rules >>>>>>>>> directly, a >>>>>>>>> user can express >>>>>>>>> their pipeline as logical flows and not worry about the >>>>>>>>> intricacies of >>>>>>>>> openflow syntax. >>>>>>>>> It does add another layer of abstraction for sure. > > > With maintenance cost in mind, we definitively prefer hosting this in > the main repository, and I think it fits nicely in there. > > -- > Frode Nordahl > >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Numan >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I welcome your thoughts and would like to know if other OVN >>>>>>>>>>> users >>>>> have >>>>>>>>>>> similar requirements. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> A proof-of-concept is available here: >>>>>>>>>>> >>>>> https://github.com/numansiddique/ovn/tree/provider_controller_support. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If there is a consensus in pursuing this further, I'll work on >>>>>>>>>>> refining the patches and submit them as RFC to start with. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> In general I have no objections to an RFC. (CC the rest of the >>>>> maintainers). >>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Numan >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Dumitru >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Ales >>>>>>> >>>>> >>>> Thanks, Dumitru _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
