Hi, Ryan, Thanks for your comments; reply inline. - Hui
Ryan Moats/Omaha/IBM wrote on 06/24/2016 08:12:38 AM: > From: Ryan Moats/Omaha/IBM > To: Hui Kang/Watson/IBM@IBMUS, "Ben Pfaff" <b...@ovn.org>, "Justin > Pettit" <jpet...@ovn.org>, "Yusheng Wang" <yshw...@vmware.com> > Cc: dev@openvswitch.org > Date: 06/24/2016 08:12 AM > Subject: Re: [ovs-dev] [OVN] Potential scalability bug in ovn-northd > on creating and binding large number of lports > > "dev" <dev-boun...@openvswitch.org> wrote on 06/23/2016 12:56:59 PM: > > > From: Hui Kang/Watson/IBM@IBMUS > > To: dev@openvswitch.org > > Date: 06/23/2016 12:57 PM > > Subject: [ovs-dev] [OVN] Potential scalability bug in ovn-northd on > > creating and binding large number of lports > > Sent by: "dev" <dev-boun...@openvswitch.org> > > > > > > Hi, > > In our scalability test for OVN, we observed an in-scalable behaviour of > > the > > ovn-northd process: the time binding a logical port increases as # of large > > port increasing, regardless of whether logical ports belong to the same > > logical > > switch. The most suspicious function in causing this issue is build_ports() > > called by ovnnb_db_run() [1], as described below. > > > > Test description: > > step 1: Create 6 logical switches. For each logical switch, create 200 > > logical ports. > > step 2: Bind 200 lports from each logical switch on an OVN chassis. > > > > Test results for step 2: > > > > # of ports | # of ovn_ports | Cpu cycle spent in | > > | allocated in build_port() | built_port(), in million | > > 200 | 200 | 25 | > > 400 | 400 | 50 | > > 600 | 600 | 75 | > > 800 | 800 | 93 | > > 1000 | 1000 | 108 | > > 1200 | 1200 | 125 | > > > > We see that on binding each logical port on a hypervisor, > > join_logical_ports() > > in build_port allocates the number of (struct ovn_port) for all the > > existing > > ports in the southbound database [2], which causes the accumulated CPU > > cycles. > > > > My question is whether there is any particular reason to allocate that > > number > > of (struct ovn_port)? It seems to me there is room in this code to optimize > > for performance. Thanks. > > > > - Hui > > > > > > [1] > > https://github.com/openvswitch/ovs/blob/master/ovn/northd/ovn-northd.c#L2529 > > [2] > > https://github.com/openvswitch/ovs/blob/master/ovn/northd/ovn-northd.c#L571 > > Hui, ovn-northd's current design is that it processes the entire ovn nb db > each computational cycle, so I would expect to see what you are seeing, which > is the argument for converting ovn-northd to an incremental processing model. > As shown in the above table, in each computation loop (especially the ovnnb_db_run), there are N (i.e., the number of all logical ports in the southbound db) is created [3]. IMO, this is unnecessary at least for the binding operation, because binding a logical port impacts a limited number of ports (e.g., lports in the same logical switch). So for incremental processing model, I think we can created a global ovn_port list and only update the ports that is triggered by the ovsdb events. [3] https://github.com/openvswitch/ovs/blob/master/ovn/northd/ovn-northd.c#L571 > Ben, Justin, Yusheng can one of you talk to an ETA for when the nlog > ovn-northd code base will start to land in the review queue? That will > provide input on whether doing an interim patch series is worth the effort > or not... What does ETA mean? I am not sure if you have read the thread about Yusheng's inital datalog patch[4] My understanding is that (1) it requires a radical redesign of the existing ovn-northd code; (2) the performance improvement is unclear. Really, the majority of the benefits of nlog comes from the feature of this programming language. Therefore, an interim patch fixing the current northd scalability issue makes sense to me. [4] http://openvswitch.org/pipermail/dev/2016-June/073176.html > > Ryan _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
