By our testing, ryu openflow has greatly improved the performance, with 500 port vxlan flow table, from 15s to 2.5s, 6 times better. ________________________________________ From: IWAMOTO Toshihiro [iwam...@valinux.co.jp] Sent: Monday, January 25, 2016 5:08 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Neutron] OVS flow modification performance
At Thu, 21 Jan 2016 02:59:16 +0000, Wuhongning wrote: > > I don't think 400 flows can show the difference , do you have setup any > tunnel peer? > > In fact we may set the network type as "vxlan", then make a fake MD simulate > sending l2pop fdb add messages, to push ten's of thousands flows into the > testing ovs agent. I chose this method because I didn't want to write such extra code for measurements. ;) Of course, I'd love to see data from other test environments and other workload than agent restarts. Also, we now have https://review.openstack.org/#/c/271939/ and can profile neutron-server (and probably others, too). I couldn't find non-trivial findings until now, though. > ________________________________________ > From: IWAMOTO Toshihiro [iwam...@valinux.co.jp] > Sent: Monday, January 18, 2016 4:37 PM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [Neutron] OVS flow modification performance > > At Mon, 18 Jan 2016 00:42:32 -0500, > Kevin Benton wrote: > > > > Thanks for doing this. A couple of questions: > > > > What were your rootwrap settings when running these tests? Did you just > > have it calling sudo directly? > > I used devstack's default, which runs root_helper_daemon. > > > Also, you mention that this is only ~10% of the time spent during flow > > reconfiguration. What other areas are eating up so much time? > > > In another run, > > $ for f in `cat tgidlist.n2`; do echo -n $f; opreport -n tgid:$f --merge > tid|head -1|tr -d '\n'; (cd bg; opreport -n tgid:$f --merge tid|head > -1);echo; done|sort -nr -k +2 > 10071 239058 100.000 python2.7 14922 100.000 python2.7 > 9995 92328 100.000 python2.7 11450 100.000 python2.7 > 7579 88202 100.000 python2.7 (18596) > 11094 51560 100.000 python2.7 47964 100.000 python2.7 > 7035 49687 100.000 python2.7 40678 100.000 python2.7 > 11093 49380 100.000 python2.7 36004 100.000 python2.7 > (legend: <pid> <oprof count with an agent restart> <junk> <junk> > <background (oprof count without an agent restart)>) > > These processes are neutron-server, nova-api, > neutron-openvswitch-agent, nova-conductor, dstat and nova-conductor in > a decending order. > > So neutron-server uses about 3x CPU time than the ovs agent, > nova-api's CPU usage is similar to the ovs agent's, and the others > aren't probably significant. > > > Cheers, > > Kevin Benton > > > > On Sun, Jan 17, 2016 at 10:12 PM, IWAMOTO Toshihiro <iwam...@valinux.co.jp> > > wrote: > > > > > I'm sending out this mail to share the finding and discuss how to > > > improve with those interested in neutron ovs performance. > > > > > > TL;DR: The native of_interface code, which has been merged recently > > > and isn't default, seems to consume less CPU time but gives a mixed > > > result. I'm looking into this for improvement. > > > > > > * Introduction > > > > > > With an ML2+ovs Neutron configuration, openflow rule modification > > > happens often and is somewhat a heavy operation as it involves > > > exec() of the ovs-ofctl command. > > > > > > The native of_interface driver doesn't use the ovs-ofctl command and > > > should have less performance impact on the system. This document > > > tries to confirm this hypothesis. > > > > > > > > > * Method > > > > > > In order to focus on openflow rule operation time and avoid noise from > > > other operations (VM boot-up, etc.), neutron-openvswitch-agent was > > > restarted and the time it took to reconfigure the flows was measured. > > > > > > 1. Use devstack to start a test environment. As debug logs generate > > > considable amount of load, ENABLE_DEBUG_LOG_LEVEL was set to false. > > > 2. Apply https://review.openstack.org/#/c/267905/ to enable > > > measurement of flow reconfiguration times. > > > 3. Boot 80 m1.nano instances. In my setup, this generates 404 br-int > > > flows. If you have >16G RAM, more could be booted. > > > 4. Stop neutron-openvswitch-agent and restart with --run-once arg. > > > Use time, oprofile, and python's cProfile (use --profile arg) to > > > collect data. > > > > > > * Results > > > > > > Execution time (averages of 3 runs): > > > > > > native 28.3s user 2.9s sys 0.4s > > > ovs-ofctl 25.7s user 2.2s sys 0.3s > > > > > > ovs-ofctl runs faster and seems to use less CPU, but the above doesn't > > > count in execution time of ovs-ofctl. > > > > > > Oprofile data collected by running "operf -s -t" contain the > > > information. > > > > > > With of_interface=native config, "opreport tgid:<pid of ovs agent>" shows: > > > > > > samples| %| > > > ------------------ > > > 87408 100.000 python2.7 > > > CPU_CLK_UNHALT...| > > > samples| %| > > > ------------------ > > > 69160 79.1232 python2.7 > > > 8416 9.6284 vmlinux-3.13.0-24-generic > > > > > > and "opreport --merge tgid" doesn't show ovs-ofctl. > > > > > > With of_interface=ovs-ofctl, "opreport tgid:<pid of ovs agent>" shows: > > > > > > samples| %| > > > ------------------ > > > 62771 100.000 python2.7 > > > CPU_CLK_UNHALT...| > > > samples| %| > > > ------------------ > > > 49418 78.7274 python2.7 > > > 6483 10.3280 vmlinux-3.13.0-24-generic > > > > > > and "opreport --merge tgid" shows CPU consumption by ovs-ofctl > > > > > > 35774 3.5979 ovs-ofctl > > > CPU_CLK_UNHALT...| > > > samples| %| > > > ------------------ > > > 28219 78.8813 vmlinux-3.13.0-24-generic > > > 3487 9.7473 ld-2.19.so > > > 2301 6.4320 ovs-ofctl > > > > > > Comparing 87408 (native python) with 62771+35774, the native > > > of_interface uses 0.4s less CPU time overall. > > > > > > * Conclusion and future steps > > > > > > The native of_interface uses slightly less CPU time but takes longer > > > time to complete a flow reconfiguration after an agent restart. > > > > > > As an OVS agent accounts for only 1/10th of total CPU usage during a > > > flow reconfiguration (data not shown), there may be other areas for > > > improvement. > > > > > > The cProfile Python module gives more fine grained data, but no > > > apparent performance bottleneck was found. The data show more > > > eventlet context switches with the native of_interface, which is due > > > to how the native of_interface is written. I'm looking into for > > > improving CPU usage and latency. > > > > > > > > > > > > __________________________________________________________________________ > > > OpenStack Development Mailing List (not for usage questions) > > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > -- > > Kevin Benton > > [1.2 <text/html; UTF-8 (quoted-printable)>] > > [2 <text/plain; us-ascii (7bit)>] > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
