I'm running NAT with iptables on server M2, and two linux PCs, M1 and M3, are directly connected to M2 as below. All interfaces are 1G NICs over 1G links.
[ M1 ]--------p1p3-[ M2 ]-p1p4--------[ M3 ] Without KNI nor DPDK, native iptables in Linux supports 360Mbps for UDP and 545Mbps for TCP. With KNI, it drops almost every packet for UDP (i.e. 87% of received packets are dropped), and shows 42Mbps for TCP. If I use KNI with DPDK, it runs in poll-mode thus there should not be interrupts. In my experiment, something strange is ksoftirqd consumes 50% CPU loads with KNI. Why ksoftirqd works such hard? I'm using the KNI application in the example directory of DPDK 2.1.0 source tree with options -c 0xf0 -n 4 -- -p 0x3 -P --config="(0,4,6,8),(1,5,7,9)". (i.e. rte_kni.ko is loaded with kthread_mode=multiple option) Any comments is appreciated. ##### without KNI (i.e. with native iptables) ##### [mslee at localhost ~]$ iperf3 -s -i 10 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48099 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 54364 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 376 MBytes 315 Mbits/sec 0.002 ms 471622/3548257 (13%) [ 5] 10.00-10.04 sec 1.53 MBytes 326 Mbits/sec 0.003 ms 1520/14071 (11%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.04 sec 435 MBytes 363 Mbits/sec 0.003 ms 473142/3562328 (13%) ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48100 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48101 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 650 MBytes 545 Mbits/sec [ 5] 10.00-10.04 sec 2.53 MBytes 563 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-10.04 sec 653 MBytes 546 Mbits/sec 97 sender [ 5] 0.00-10.04 sec 652 MBytes 545 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ##### with KNI (i.e. with KNI + native iptables) ##### [mslee at localhost ~]$ iperf3 -s -i 10 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48102 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 60867 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 57.9 MBytes 48.6 Mbits/sec 0.021 ms 3239121/3713340 (87%) [ 5] 10.00-10.27 sec 278 KBytes 8.37 Mbits/sec 0.018 ms 16319/18541 (88%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.27 sec 456 MBytes 372 Mbits/sec 0.018 ms 3255440/3731881 (87%) ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48103 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48104 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 50.6 MBytes 42.4 Mbits/sec [ 5] 10.00-10.04 sec 204 KBytes 44.2 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-10.04 sec 52.0 MBytes 43.4 Mbits/sec 151 sender [ 5] 0.00-10.04 sec 50.8 MBytes 42.4 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ##### snapshot of top command ##### PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9519 root 20 0 8659492 20404 19948 R 400.0 0.1 10:41.09 unatd-kni 114 root 20 0 0 0 0 S 45.7 0.0 1:31.46 ksoftirqd/8 9525 root 20 0 0 0 0 S 17.2 0.0 0:07.89 kni_p1p3 9590 root 20 0 0 0 0 S 1.2 0.0 0:01.57 kni_p1p4 -- Moon-Sang Lee, SW Engineer Email: sang0627 at gmail.com Wisdom begins in wonder. *Socrates*
