Ryan Moats/Omaha/IBM wrote on 06/24/2016 10:51:10 PM:
> From: Ryan Moats/Omaha/IBM > To: Hui Kang/Watson/IBM@IBMUS > Cc: dev@openvswitch.org > Date: 06/24/2016 10:51 PM > Subject: Re: [ovs-dev] [OVN] Potential scalability bug in ovn-northd > on creating and binding large number of lports > > "dev" <dev-boun...@openvswitch.org> wrote on 06/23/2016 12:56:59 PM: > > > From: Hui Kang/Watson/IBM@IBMUS > > To: dev@openvswitch.org > > Date: 06/23/2016 12:57 PM > > Subject: [ovs-dev] [OVN] Potential scalability bug in ovn-northd on > > creating and binding large number of lports > > Sent by: "dev" <dev-boun...@openvswitch.org> > > > > > > Hi, > > In our scalability test for OVN, we observed an in-scalable behaviour of > > the > > ovn-northd process: the time binding a logical port increases as # of large > > port increasing, regardless of whether logical ports belong to the same > > logical > > switch. The most suspicious function in causing this issue is build_ports() > > called by ovnnb_db_run() [1], as described below. > > > > Test description: > > step 1: Create 6 logical switches. For each logical switch, create 200 > > logical ports. > > step 2: Bind 200 lports from each logical switch on an OVN chassis. > > > > Test results for step 2: > > > > # of ports | # of ovn_ports | Cpu cycle spent in | > > | allocated in build_port() | built_port(), in million | > > 200 | 200 | 25 | > > 400 | 400 | 50 | > > 600 | 600 | 75 | > > 800 | 800 | 93 | > > 1000 | 1000 | 108 | > > 1200 | 1200 | 125 | > > > > We see that on binding each logical port on a hypervisor, > > join_logical_ports() > > in build_port allocates the number of (struct ovn_port) for all the > > existing > > ports in the southbound database [2], which causes the accumulated CPU > > cycles. > > > > My question is whether there is any particular reason to allocate that > > number > > of (struct ovn_port)? It seems to me there is room in this code to optimize > > for performance. Thanks. > > > > - Hui > > > > > > [1] > > https://github.com/openvswitch/ovs/blob/master/ovn/northd/ovn-northd.c#L2529 > > [2] > > https://github.com/openvswitch/ovs/blob/master/ovn/northd/ovn-northd.c#L571 > > Hui- > > Since it looks like simple short term optimization of northd is a > "Good Thing" (TM) [1], Is the above the "long pole in the tent" or > are there other hot spots of similar brightness? > > I'm planning at looking at what might be possible, and want to know if > there are other spots that should be included in the pass. Hi, Ryan, Thanks for your attention to this problem. As of now, our profiling data shows that the build_ports (including its callee such as join_logical_port) is the hot spot. We can discuss on IRC or slack about how to optimize. Regards, - Hui > > Thanks in advance, > Ryan > > [1] http://openvswitch.org/pipermail/dev/2016-June/073574.html _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
