On Wed, Jun 18, 2025 at 5:00 PM Dumitru Ceara <dce...@redhat.com> wrote: > > Hi Han, > > On 6/18/25 7:40 PM, Han Zhou wrote: > > Thanks Numan for the proposal. > > > > On Wed, Jun 18, 2025 at 4:27 AM Dumitru Ceara <dce...@redhat.com> wrote: > >> > >> Hi Numan, Ales, > >> > >> On 6/10/25 4:24 PM, Numan Siddique wrote: > >>> On Tue, Jun 10, 2025 at 2:04 AM Ales Musil <amu...@redhat.com> wrote: > >>>> > >>>> > >>>> > >>>> On Thu, Jun 5, 2025 at 9:44 PM Numan Siddique <num...@ovn.org> wrote: > >>>>> > >>>>> On Thu, Jun 5, 2025 at 11:33 AM Dumitru Ceara <dce...@redhat.com> > > wrote: > >>>>>> > >>>>>> On 6/4/25 6:24 PM, Numan Siddique wrote: > >>>>>>> Hello, > >>>>>>> > >>>>>> > >>>>>> Hi Numan, > >>>> > >>>> > >>>> Hi Dumitru and Numan, > >>>> > >>>> I think the project might be a good idea in general, just adding > >>>> my opinion on some parts below. > >>>> > >>>>>> > >>>>>>> There's a need to configure the provider bridge with specific > > OpenFlow > >>>>>>> rules after packets leave the OVN pipeline and enter via the patch > >>>>>>> port. > >>>>>>> > >>>>>>> To simplify this for CMS, I propose utilizing OVN logical flows. > > This > >>>>>>> would eliminate the need for CMS to manage direct OpenFlow > > connections > >>>>>>> and programming. > >>>>>>> > >>>>>> > >>>>>> This seems very interesting! > >>>>>> > >>>>>>> To achieve this, I've developed a new service within OVN called > >>>>>>> `ovn-pr-controller` (pr = provider). Here's a high-level overview: > >>>>>>> > >>>>>>> A new database, `OVN_Provider`, is created with two main tables: > >>>>>>> `PR_Bridge` and `Logical_Flow`. > >>>>>>> > >>>>>>> `ovn-pr-controller` connects to this database, translates logical > >>>>>>> flows to OpenFlow rules, and programs the bridges. > >>>>>>> > >>>>>>> CMS adds logical flows for managed provider bridges by connecting to > >>>>>>> the `OVN_Provider` database. > >>>>>>> > >>>>>>> CMS can define the pipeline as needed. > >>>>>>> > >>>>>>> An `ovn-prctl` utility (similar to `ovn-nbctl`) is used to program > >>>>>>> logical flows. > >>>>>>> > >>>>>>> Example Usage: > >>>>>>> > >>>>>>> # Add a provider bridge > >>>>>>> `ovn-prctl add-br br-ext` > >>>>>>> > >>>>>>> # Add logical flows > >>>>>>> `ovn-prctl add-flow br-ext 0 100 "inport == \\"patch-port\\"" > >>>>>>> "ct_snat_zone = 1000; next;"` > >>>>>>> `ovn-prctl add-flow br-ext 0 0 "1” "next;"` > >>>>>>> > >>>>>>> `ovn-prctl add-flow br-ext 1 1000 "ip4" "ct_snat;"` > >>>>>>> > >>>>>>> `ovn-prctl add-flow br-ext 2 1000 "ip4 && ct.new && ct.trk && > > ip4.src > >>>>>>> == 10.0.0.11" "ct_snat(100.64.0.11); next;"` > >>>>>>> > >>>>>>> `ovn-prctl add-flow br-ext 2 0 "ip4" "next;"` > >>>>>>> > >>>>>>> `ovn-prctl add-flow br-ext 3 100 "ip4 && ip4.dst == 52.92.128.0/17" > >>>>>>> "tun.id = 1000; tun_ip4.dst = 10.100.100.1; eth.dst = > >>>>>>> 4c:96:14:14:01:b0; outport = \\"vxlan0\\"; output;"` > >>>>>>> > >>>>>>> `ovn-prctl add-flow br-ext 3 0 “1” “output;”` > >>>>>>> > >>>>>>> I'd like to get the community's feedback on whether this service > > would > >>>>>>> be a valuable addition to OVN. > >>>>>>> > >>>>>> > >>>>>> I wonder if we need these to be separate databases/processes. > > Wouldn't > >>>>>> it make sense to expose this in the NB database directly? > >>>>>> > >>>>>> With your proposal, these new ovn-pr-controllers would have to > > connect > >>>>>> to the central OVN_Provider database from all chassis. But we > > already > >>>>>> have the ovn-controller connection to the Southbound. Could we > > reuse that? > >>>>> > >>>>> I forgot to mention that the OVN provider ovsdb-server and the > >>>>> ovn-pr-controller needs > >>>>> to be run on each chassis. > >>>>> > >> > >> I see. However, I think it gives even more flexibility if the OVN > >> provider database is centralized. Then the CMS central component (e.g., > >> neutron for OpenStack) could just write "logical flows" there and rely > >> on per-chassis ovn-pr-controllers to translate that to OpenFlow. > >> > > While it brings flexibility, we should also keep in mind that the central > > approach could introduce another choke point for scale. Even if we reuse SB > > DB as the central component, the new ovn-pr-controllers process would > > require new connections to the SB DB. > > > > True, but I was not really thinking of using the SB DB as the central > component for ovn-pr-controllers but rather a new database server. And > I really hope it wouldn't require storing as many logical flows (and > data in general) as we store in the SB today. >
Thanks for the comments and the discussion. Just FYI - I thought we are in favour of having a separate project. So I reworked a bit and renamed the project to - ovnbr controller (ovn bridge controller) - https://github.com/numansiddique/ovnbr/tree/initial_commits But I can move back the code to ovn project. So not a problem. > > Supporting both central and distributed may be an overkill at least as a > > start, if we could decide which one is more practical from a CMS point of > > view. > > > > Maybe, but the "distributed" case would probably mean more work for the > CMS on the node side (a per chassis CMS component is definitely required > then to provision the per chassis database). > > I do understand however that we'd need a way to abstract > chassis-specific resources (e.g., OVS ports) in the other, > "centralized", case. And that might be a significant disadvantage. > > >> On second thought, if designed properly, there's probably no need to add > >> a restriction to colocate the new database and ovn-pr-controller. We > >> could just support both types of deployments: @Dumitru Ceara If I understand correctly, what you're suggesting is that CMS can decide to run ovnbr database either central or locally - for each chassis. I think that would work too. Only disadvantage is that CMS has to ensure that the bridge names and physical interface names have to be the same. Although we can definitely find a way to handle this. The present proposal schema - https://github.com/numansiddique/ovnbr/blob/initial_commits/ovn-br.ovsschema#L66 has only a Bridge table and no ports defined. In the present proposal CMS can add logical flows like - "inport == eth1 && ....." and ovnbr-controller expects an OVS interface - eth1 in the OVS bridge. > >> - ovn-pr-controller & database on the same chassis > >> - (multiple) ovn-pr-controller connecting to a database on a different > >> node (e.g., colocated with the Southbound) > >> > >>>>> This is basically to allow a CMS agent running on each chassis to use > >>>>> logical flows > >>>>> and program them in the OVN provider database, instead of talking > >>>>> OpenFlow directly. > >>>>> > >>>>> OVN has a very nice way of abstracting OpenFlows using logical flows > >>>>> and my idea is to leverage it. > >>>>> > >>>>> ovnkube-node, as you know, programs the OpenFlow rules to the > >>>>> provider bridge directly by execing > >>>>> "ovs-ofctl". The OVN kubernetes community could possibly use OVN > >>>>> provider controller. > >> > >> Another thing the CMS node component (ovnkube-node in the example above) > >> needs to do is reconcile flows on the "provider" bridge on restart. > >> > >> This new OVN provider controller would also remove the need for the CMS > >> to do that, which is an advantage. > >> > > > > It is definitely valuable, and no objection from me. But I think it is > > better to ask opinion from CMS component maintainers if the OVS pipeline > > for the provider bridge is complex enough to justify this. Also, how much > > would it help by managing logical flows through an OVSDB versus managing > > open-flow rules though ovs-ofctl. Keep in mind that the abstraction here is > > not at the level of the NB DB which focuses on logical topologies - here > > the abstraction is logical-flows (something at the SB level), so the CMS > > still needs to manage the pipeline rather than just logical topology. That's correct. Expectation is that CMS knows what it is doing, And we have the requirement to add flows in the provider bridges and hence thought of this abstraction to - leverage the OVN logical flows - and to leverage the openflow handling of ovn-controller. My personal preference is also to have it part of OVN so that it is easier to manage. But as Ales mentioned, we have many logical actions like - put_dhcp_opts etc (which use controller action) which will not work with the ovn-bridge-controller. Probably ovn-bridge-controller can maintain its own symbol table. Thanks Numan > > > > Adding @Tim Rozet <tro...@redhat.com> @Girish Moodalbail > > <gmoodalb...@gmail.com> from ovn-k8s to comment. > > > > Adding Jakub Libosvar too, from the neutron side, as we briefly touched > on the content of this proposal in a discussion while trying to onboard > the OVN BGP support in neutron. IMO a centralized "provider network > database" might fit that specific use case quite well because the open > flow rules that needed to be added to the provider bridge were quite > generic. > > >>>>> I'm not suggesting this, but just giving an example or a potential > > user. > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> I believe it could be useful, but I'm unsure if it should be > >>>>>>> integrated into OVN directly or be a separate project within > >>>>>>> `ovn-org`. > >>>> > >>>> > >>>> In my opinion it would be better as a standalone project within > >>>> ovn-org. There is couple of reasons why: > >>>> > >>>> 1) Separation between ovn actions and the new ones. I suppose the > >>>> ovn-pr-controller won't support like put_arp etc. in other words > >>>> actions that require userspace handling. > >>>> > >>>> 2) There are certain expectations about parameters, mostly ct actions > >>>> and it feels a little strange to populate a specific register in order > >>>> to get the action working. > >>>> > >>>> 3) Maintenance cost to some extent, standalone project might be > >>>> easier to maintain, it can be written in different language if it > >>>> would be better fit. > >>>> > >>>> 4) The abstraction can be generated automatically to some extent > >>>> so most of the code might be actually a way to generate the parsing > >>>> etc. > >>>> > >>>> I'm fully aware that this would require some duplication. With that > >>>> said, this just my opinion and I won't block any other decision. > >>> > >>> > >>> Hi Ales, > >>> > >>> Thanks for the replies. I agree with your points. I think it makes > >>> sense to have > >>> a separate project even if the code is duplicated. > >>> > >> > >> I'm not sure I see anything blocking the ovn-pr-controller (or whatever > >> we decide to call it) code from being part of the ovn-org/ovn project. > >> > >> It can be a standalone daemon (similar to controller/ovn-controller or > >> controller-vtep/ovn-controller-vtep). > >> > >> It can even be written in a different language (e.g., rust) if we wish. > >> > >> Keeping it in the same project actually reduces maintenance cost because > >> we can just reuse the expr parsing library from OVN directly. We can > >> also reuse all the build/test infra (if we wish). > >> > >> It would also mean we wouldn't have to come up with different release > >> strategies, contribution guidelines, etc. > >> > >> Also, with downstream in mind, it might be (slightly) easier for distros > >> to package this new binary if its code lived in ovn-org/ovn. > >> > > > > In my option it is really not good to duplicate components in different > > repos. Unless we find ways to expose logical flow translation as a library > > and share between the projects, I am inclined to use OVN repo directly. > > > > Thanks, > > Han > > > >>> Implementing in a different language would be nice but I think it will > >>> take a lot > >>> of effort to implement expr parsing and openflow encoding. > >>> > >>> Thanks > >>> Numan > >>> > >> > >> Thanks, > >> Dumitru > >> > >>>> > >>>>>>> > >>>>>> > >>>>>> I guess it depends a bit on the approach chosen for implementation > > but > >>>>>> at a first glance this seems as a good candidate for ovn-org to me. > >>>>>> > >>>>>>> Three possible options are: > >>>>>>> > >>>>>>> - Integrate the `OVN provider controller` into `ovn-org/ovn`. > >>>>>>> > >>>>>>> - Create a separate project within `ovn-org` (which would require > >>>>>>> duplicating some files like `lib/actions.c`). > >>>>>>> > >>>>>>> - Do not pursue this. > >>>>>>> > >>>>>> > >>>>>> Listing it here because it's on the "con" side of things. :) I > > wonder > >>>>>> if we'll end up duplicating all OVS actions into OVN logical flow > > actions. > >>>>> > >>>>> Actually that's my goal. Instead of using OpenFlow rules directly, a > >>>>> user can express > >>>>> their pipeline as logical flows and not worry about the intricacies of > >>>>> openflow syntax. > >>>>> It does add another layer of abstraction for sure. > >>>>> > >>>>> Thanks > >>>>> Numan > >>>>> > >>>>> > >>>>>> > >>>>>>> I welcome your thoughts and would like to know if other OVN users > > have > >>>>>>> similar requirements. > >>>>>>> > >>>>>>> > >>>>>>> A proof-of-concept is available here: > >>>>>>> > > https://github.com/numansiddique/ovn/tree/provider_controller_support. > >>>>>>> > >>>>>>> > >>>>>>> If there is a consensus in pursuing this further, I'll work on > >>>>>>> refining the patches and submit them as RFC to start with. > >>>>>>> > >>>>>> > >>>>>> In general I have no objections to an RFC. (CC the rest of the > > maintainers). > >>>>>> > >>>>>>> Thanks, > >>>>>>> Numan > >>>>>> > >>>>>> Regards, > >>>>>> Dumitru > >>>>>> > >>>>> > >>>> > >>>> Thanks, > >>>> Ales > >>> > > > > Regards, > Dumitru > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
