This series tries to improve the kernel datapath - ofproto upcall processing code path. Some of these are local changes that may be easier to adopt, while others will require more discussion (i.e. modifications to the kernel interface).
'make check' passes at each stage, but otherwise I have tested these only as described below. Summary: The share of packets reaching the userspace flow table doubles. By far the most of the improvement comes from netlink upcall request pooling, which is a change to the dpif-provider interface. Note: The test described below almost exclusively runs on the "without_facet" code path, so it does not measure the benefit for kernel flow setup. Test protocol: - 'udpping' sends 10000000 UDP packets with a unique source and destination port pair to br0 in bursts of 97 packets. Between bursts it nanosleeps for 50 nanoseconds. This results into ~80% CPU (core) utilization. - Start test when CPU >96.5% idle - Collect 5 flow dumps after runs during which the ovs-vswitchd process remains >97.5% CPU utilization throughout the test. - Keep 3 best out of 5 collected for each test case. - Compute percentage of mean packets processed by the flow entry. Test system: Linux debian-jr 3.2.0-4-amd64 #1 SMP Debian 3.2.35-2 x86_64 GNU/Linux 2GB VM on VirtualBox 4.2.6 Host: Intel Core2 Duo 2.93 GHz OSX 10.6.8 In 1st shell: # rmmod openvswitch # ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --bootstrap-ca-cert=db:SSL,ca_cert --pidfile --detach # insmod datapath/linux/openvswitch.ko # ovs-vsctl --no-wait del-br br0 # ovs-vsctl --no-wait add-br br0 # chrt 99 ovs-vswitchd --pidfile -verr In 2nd shell: # ovs-ofctl del-flows br0 # ovs-ofctl add-flow br0 "in_port=LOCAL actions=" # nice -n -5 ./udpping Results: --- ovs git master (84507691a510efa76a0f063ee04a06cf19081ff1): n_packets=3103133 n_packets=3085670 n_packets=3274467 Mean: 3154423 = 31.5 % --- 01/11: dpif: Take upcall netlink pooling down to the system call interface. n_packets=5540243 n_packets=5804791 n_packets=5782079 Mean: 5709038 = 57.1 % --- 02/11: netlink-socket: Use recvmmsg() system call with kernels since 2.6.32. Note: This may be Linux version dependent. n_packets=5568893 n_packets=5897427 n_packets=5893462 Mean: 5786594 = 57.9 % --- 03/11: datapath: Fold key_len and hash fields into struct sw_flow_key. Note: This patch saves unnecessary hash calculations on kernel datapath flow setup. This test does not stress that code path. n_packets=5946011 n_packets=5855520 n_packets=5936731 Mean: 5912754 = 59.1 % --- 06/11: ofproto-dpif: Deferred setting of packet layer pointers. n_packets=6053026 n_packets=6005442 n_packets=5997440 Mean: 6018636 = 60.2 % --- 09/11: datapath: Pass more packet metadata to userspace. (dpif-linux: Use kernel provided layer pointers) n_packets=5937665 n_packets=6354151 n_packets=6195069 Mean: 6162295 = 61.6 % --- 10/11: ofproto-dpif: Use kernel provided flow hash in upcall pooling. Note: Hashes are also used when creating facets and subfacets. This test does not stress that code path. n_packets=6430066 n_packets=6416697 n_packets=6071671 Mean: 6306144 = 63.1 % --- Jarno Rajahalme (11): dpif: Pool dpif_recv() calls. netlink-socket: Use recvmmsg() with kernels since 2.6.32. datapath: Fold key_len and hash into sw_flow_key. lib/flow: Split flow_extract(). ofproto-dpif: Collect TCP flags only when needed. ofproto-dpif: Deferred setting of packet layer pointers. lib/packets: Remove the unused l7 setting for ICMP. dpif: Prepare for additional kernel provided packet metadata. datapath: Pass more packet metadata to userspace. ofproto-dpif: Use kernel provided flow hash in upcall pooling. dpif-netdev: pass layer pointers and key hash like dpif-linux. datapath/datapath.c | 61 ++++++++------ datapath/flow.c | 176 +++++++++++++++++++++------------------ datapath/flow.h | 29 ++++--- datapath/vport-vxlan.c | 2 +- include/linux/openvswitch.h | 10 +++ lib/dpif-linux.c | 161 +++++++++++++++++++++++++----------- lib/dpif-netdev.c | 56 ++++++++++--- lib/dpif-provider.h | 32 +++++--- lib/dpif.c | 57 ++++++++----- lib/dpif.h | 6 +- lib/flow.c | 42 +++++++--- lib/flow.h | 1 + lib/netlink-socket.c | 96 ++++++++++++++++++++++ lib/netlink-socket.h | 11 +++ lib/odp-util.c | 16 ++++ lib/odp-util.h | 3 +- lib/packets.c | 44 +++++----- lib/packets.h | 23 +++++- ofproto/in-band.c | 30 ++++--- ofproto/ofproto-dpif.c | 192 ++++++++++++++++++++++++++----------------- 20 files changed, 720 insertions(+), 328 deletions(-) -- 1.7.10.4 _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
