Wondering if those experienced with other netmap drivers might be able to comment what is limiting performance of mlx4. It seems that the reason pkt-gen is only getting 2.4Mpps with mlx4 40G is that pkt-gen is saturating a core. This clearly shouldn’t be the case as evidenced by netmap papers (14.8Mpps at 900Mz core). As would be expected, the output from ‘perf top’ shows that sender_body and poll() are the largest userspace CPU hogs (measured in % of samples—over 24 cpus)
29.65% [netmap] [k] netmap_poll 12.47% [mlx4_en] [k] mlx4_netmap_txsync 8.69% libc-2.19.so [.] poll 6.15% pkt-gen [.] sender_body 2.26% [kernel] [k] local_clock 2.12% [kernel] [k] context_tracking_user_exit 1.87% [kernel] [k] select_estimate_accuracy 1.81% [kernel] [k] system_call …. 1.24% [netmap] [k] nm_txsync_prologue …. 0.63% [mlx4_en] [k] mlx4_en_arm_cq 0.61% [kernel] [k] account_user_time Furthermore it appears from annotating the code in pkt-gen.c with utilization, about 50% of sender_body is spent on this line while iterating through the rings: https://github.com/caldweba/netmap/blob/master/examples/pkt-gen.c#L1091 <https://github.com/caldweba/netmap/blob/master/examples/pkt-gen.c#L1091> if (nm_ring_empty(txring)) Does this mean it is waiting for free slots most of the time and increasing from 8 rings might help? Here are the current module parameters in case they shed light on the issue. Also, netmap config kernel messages are shown below. Thanks in advance. /sys/module/netmap/parameters/adaptive_io: 0 /sys/module/netmap/parameters/admode: 0 /sys/module/netmap/parameters/bridge_batch: 1024 /sys/module/netmap/parameters/buf_curr_num: 163840 /sys/module/netmap/parameters/buf_curr_size: 2048 /sys/module/netmap/parameters/buf_num: 163840 /sys/module/netmap/parameters/buf_size: 2048 /sys/module/netmap/parameters/default_pipes: 0 /sys/module/netmap/parameters/flags: 0 /sys/module/netmap/parameters/fwd: 0 /sys/module/netmap/parameters/generic_mit: 100000 /sys/module/netmap/parameters/generic_rings: 1 /sys/module/netmap/parameters/generic_ringsize: 1024 /sys/module/netmap/parameters/if_curr_num: 100 /sys/module/netmap/parameters/if_curr_size: 1024 /sys/module/netmap/parameters/if_num: 100 /sys/module/netmap/parameters/if_size: 1024 /sys/module/netmap/parameters/mitigate: 1 /sys/module/netmap/parameters/mmap_unreg: 0 /sys/module/netmap/parameters/no_pendintr: 1 /sys/module/netmap/parameters/no_timestamp: 0 /sys/module/netmap/parameters/priv_buf_num: 4098 /sys/module/netmap/parameters/priv_buf_size: 2048 /sys/module/netmap/parameters/priv_if_num: 1 /sys/module/netmap/parameters/priv_if_size: 1024 /sys/module/netmap/parameters/priv_ring_num: 4 /sys/module/netmap/parameters/priv_ring_size: 20480 /sys/module/netmap/parameters/ring_curr_num: 200 /sys/module/netmap/parameters/ring_curr_size: 36864 /sys/module/netmap/parameters/ring_num: 200 /sys/module/netmap/parameters/ring_size: 36864 /sys/module/netmap/parameters/txsync_retry: 2 /sys/module/netmap/parameters/verbose: 0 > On May 28, 2015, at 12:47 AM, Blake Caldwell <caldw...@colorado.edu> wrote: > > Hello, > > I have made necessary tweaks to the mlx4 patches for a successful build on > Linux 3.13.11 (Ubuntu 14.04) and enabled the driver in the Linux build > system. See: > https://github.com/caldweba/netmap.git > <https://github.com/caldweba/netmap.git> for my additional commits. > > Without any core modifications to the mlx4 netmap driver, I am actually > getting reduced performance, 2.5 Mpps on a 40G port. I’m interested in > improving the performance of this driver, but as I’m new to netmap and even > these drivers, some assistance would be welcome. As Luigi mentioned, the > Mellanox developer documentation seems to be a stumbling point. Would anyone > from Mellanox be able to lend some expertise? > > It would appear mlx4_netmap_txsync() is the place to focus optimization, and > the comments Luigi put in will be helpful. Although I’m a little confused the > on the remaining work for mlx4_netmap_tx_config (marked TODO). See > https://github.com/caldweba/netmap/blob/master/LINUX/mlx4_netmap_linux.h > <https://github.com/caldweba/netmap/blob/master/LINUX/mlx4_netmap_linux.h> > for Luigi’s current mlx4_netmap_txsync() code. > > Below is my output from pkt-gen and from ethtool on the device. > > Best regards, > Blake > > ————— > $ sudo build-apps/pkt-gen -i p2p1 -f tx -n 500111222 -l 60 -w 5 > 060.428278 main [1649] interface is p2p1 > 060.428770 extract_ip_range [287] range is 10.0.0.1:0 to 10.0.0.1:0 > 060.428782 extract_ip_range [287] range is 10.1.0.1:0 to 10.1.0.1:0 > 060.875064 main [1840] mapped 334980KB at 0x7fd1f04d5000 > Sending on netmap:p2p1: 8 queues, 1 threads and 1 cpus. > 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff) > 060.875151 main [1924] Sending 512 packets every 0.000000000 s > 060.875158 main [1926] Wait 5 secs for phy reset > 065.875244 main [1928] Ready... > 065.875276 nm_open [456] overriding ifname p2p1 ringid 0x0 flags 0x1 > 065.914805 sender_body [1014] start, fd 4 main_fd 3 > 065.958284 sender_body [1083] drop copy > 066.915788 main_thread [1446] 2468560 pps (2471088 pkts in 1001024 usec) > 067.916827 main_thread [1446] 2476292 pps (2478865 pkts in 1001039 usec) > 068.917815 main_thread [1446] 2476261 pps (2478708 pkts in 1000988 usec) > 069.918864 main_thread [1446] 2476232 pps (2478827 pkts in 1001048 usec) > 070.919902 main_thread [1446] 2476031 pps (2478604 pkts in 1001039 usec) > 071.920920 main_thread [1446] 2476304 pps (2478825 pkts in 1001018 usec) > 072.921896 main_thread [1446] 2476349 pps (2478766 pkts in 1000976 usec) > 073.922948 main_thread [1446] 2476327 pps (2478932 pkts in 1001052 usec) > 074.923924 main_thread [1446] 2476301 pps (2478715 pkts in 1000975 usec) > 075.924903 main_thread [1446] 2476257 pps (2478681 pkts in 1000979 usec) > 076.925918 main_thread [1446] 2476195 pps (2478708 pkts in 1001015 usec) > 077.926970 main_thread [1446] 2476242 pps (2478849 pkts in 1001053 usec) > > dmesg: > [52591.017469] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb > 2014) > [52591.017621] mlx4_en 0000:04:00.0: registered PHC clock > [52591.017780] mlx4_en 0000:04:00.0: Activating port:1 > [52591.023552] mlx4_en: eth0: Using 192 TX rings > [52591.023554] mlx4_en: eth0: Using 8 RX rings > [52591.023556] mlx4_en: eth0: frag:0 - size:1526 prefix:0 align:0 > stride:1536 > [52591.040585] mlx4_en: eth0: Initializing port > [52591.040732] 779.121252 [2720] netmap_attach success for eth0 > tx 8/512 rx 8/1024 queues/slots > [52591.060580] mlx4_en 0000:04:00.0: Activating port:2 > [52591.068337] mlx4_en: eth1: Using 192 TX rings > [52591.068340] mlx4_en: eth1: Using 8 RX rings > [52591.068342] mlx4_en: eth1: frag:0 - size:1526 prefix:0 align:0 > stride:1536 > [52591.085696] mlx4_en: eth1: Initializing port > [52591.085867] 779.166352 [2720] netmap_attach success for eth1 > tx 8/512 rx 8/1024 queues/slots > [52591.960730] mlx4_en: eth0: Link Up > [52593.029536] systemd-udevd[50993]: renamed network interface eth0 to p2p1 > [52593.061736] systemd-udevd[50996]: renamed network interface eth1 to > rename28 > [52624.680481] mlx4_en: p2p1: frag:0 - size:1526 prefix:0 align:0 > stride:1536 > [52624.834109] 812.888289 [ 473] mlx4_en_tx_irq XXXXXXXXX tx_irq > 0 unexpected, ignoring > > [55436.322304] 622.179577 [ 665] mlx4_netmap_config using only 8 out > of 192 tx queues > [55436.339688] 622.196947 [ 672] mlx4_netmap_config txr 8 txd 512 > bufsize 32768 -- rxr 8 rxd 1024 act 1024 bufsize 16384 > [55436.361877] 622.219119 [ 124] mlx4_netmap_reg setting netmap > mode for eth0 to ON > [55436.379345] 622.236575 [ 127] mlx4_netmap_reg unloading eth0 > [55436.485781] 622.342926 [ 163] mlx4_netmap_reg loading eth0 > [55436.501124] mlx4_en: p2p1: frag:0 - size:1526 prefix:0 align:0 > stride:1536 > [55436.517514] 622.374635 [ 628] mlx4_netmap_rx_config stride 16 possible > frags 1 descsize 0 DS_SIZE 16 > [55436.536462] 622.393570 [ 648] mlx4_netmap_rx_config ring 0 done > [55436.551746] 622.408842 [ 648] mlx4_netmap_rx_config ring 1 done > [55436.567111] 622.424194 [ 628] mlx4_netmap_rx_config stride 16 possible > frags 1 descsize 0 DS_SIZE 16 > [55436.586261] 622.443330 [ 648] mlx4_netmap_rx_config ring 2 done > [55436.601844] 622.458900 [ 648] mlx4_netmap_rx_config ring 3 done > [55436.617525] 622.474569 [ 648] mlx4_netmap_rx_config ring 4 done > [55436.633057] 622.490089 [ 648] mlx4_netmap_rx_config ring 5 done > [55436.648376] 622.505396 [ 648] mlx4_netmap_rx_config ring 6 done > [55436.780501] 622.637414 [ 165] mlx4_netmap_reg start_port returns > 0 > [55436.796403] mlx4_en: p2p1: Link Down > [55437.755281] mlx4_en: p2p1: Link Up > > > $ ethtool p2p1 > Settings for p2p1: > Supported ports: [ TP ] > Supported link modes: 10000baseT/Full > Supported pause frame use: No > Supports auto-negotiation: No > Advertised link modes: 10000baseT/Full > Advertised pause frame use: No > Advertised auto-negotiation: No > Speed: 40000Mb/s > Duplex: Full > Port: Twisted Pair > PHYAD: 0 > Transceiver: internal > Auto-negotiation: off > MDI-X: Unknown > Cannot get wake-on-lan settings: Operation not permitted > Current message level: 0x00000014 (20) > link ifdown > Link detected: yes > > $ ethtool -i p2p1 > driver: mlx4_en > version: 2.2-1 (Feb 2014) > firmware-version: 2.30.3200 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: no > supports-priv-flags: yes > > ethtool -g p2p1 > Ring parameters for p2p1: > Pre-set maximums: > RX: 8192 > RX Mini: 0 > RX Jumbo: 0 > TX: 8192 > Current hardware settings: > RX: 1024 > RX Mini: 0 > RX Jumbo: 0 > TX: 512 > > Coalesce parameters for p2p1: > Adaptive RX: on TX: off > stats-block-usecs: 0 > sample-interval: 0 > pkt-rate-low: 400000 > pkt-rate-high: 450000 > > rx-usecs: 16 > rx-frames: 44 > rx-usecs-irq: 0 > rx-frames-irq: 0 > > tx-usecs: 16 > tx-frames: 16 > tx-usecs-irq: 0 > tx-frames-irq: 0 > > rx-usecs-low: 0 > rx-frame-low: 0 > tx-usecs-low: 0 > tx-frame-low: 0 > > rx-usecs-high: 128 > rx-frame-high: 0 > tx-usecs-high: 0 > tx-frame-high: 0 > > $ sudo ethtool -k p2p1 > Features for p2p1: > rx-checksumming: on > tx-checksumming: on > tx-checksum-ipv4: on > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: on > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: on > tx-scatter-gather: on > tx-scatter-gather-fraglist: off [fixed] > tcp-segmentation-offload: on > tx-tcp-segmentation: on > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: on > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: on > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: on > tx-vlan-offload: on > ntuple-filters: off [fixed] > receive-hashing: on > highdma: on [fixed] > rx-vlan-filter: on [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > tx-mpls-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: on > loopback: off > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > > >> On May 20, 2015, at 9:18 AM, Luigi Rizzo <ri...@iet.unipi.it >> <mailto:ri...@iet.unipi.it>> wrote: >> >> hi all, >> >> the mlx4 netmap patch (for linux only) was something i did long >> ago when i had some mellanox hardware available, but no documentation >> so i had to resort to interpreting what the linux driver did. >> >> At the time i had the following performance (on PCIe v2 bus): >> >> 10G ports: tx/rx at about 7 Mpps with 64 byte packets >> could saturate the link with 192 or 256 byte packets >> >> 40G ports: tx/rx at about 11 Mpps with 64 byte packets >> max 28 Gbit/s even with 1500 byte frames >> >> I don't know if the limited performance was due to bus, >> firmware or lack of documentation, anyways this is not >> something i can or want to deal with. >> >> My understanding is that Mellanox does not release programming >> documentation, so the only way to have native netmap support >> for that card would be to have Mellanox work on that and >> provide a suitable patch. >> >> I do not expect more than a week's work (the typical extra >> code in each driver is about 500 lines, and very simple) >> for someone with access to documentation. Also, the patch >> for FreeBSD and Linux is typically very similar so once we >> have a driver for one, the other would be trivial. >> >> It would be of course great to add Mellanox to the list of >> devices with native netmap support, together with Chelsio >> and Intel. >> >> Perhaps Hans (who may have contacts) can talk to the right >> people and figure out. On my side, I am happy to give directions >> on what needs to be done and import any patch that should >> be made available. >> >> cheers >> luigi >> >> On Wed, May 20, 2015 at 4:50 PM, Hans Petter Selasky <h...@selasky.org >> <mailto:h...@selasky.org>> wrote: >> On 05/20/15 16:13, Blake Caldwell wrote: >> Hello, >> >> I noticed that the mlx4_en patch for netmap is LINUX/wip-patches, so they >> are not enabled in the normal build process. I’m curious about the status of >> mlx4 support? >> >> If additional work to the patches is needed, any details as to what the >> issues were. >> >> Any info would be great! Thanks in advance! >> >> >> Hi Blake, >> >> The MLX4 driver is being actively maintained in -stable and -current. >> Regarding netmap support for the FreeBSD MLX4 en driver, I'm not sure. Maybe >> Oded knows, CC'ed? Do you have a link for the patch you are referring? >> >> This there any particular use-case you are interested in? >> >> --HPS >> >> >> _______________________________________________ >> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> <http://lists.freebsd.org/mailman/listinfo/freebsd-net> >> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org >> <mailto:freebsd-net-unsubscr...@freebsd.org>" >> >> >> >> -- >> -----------------------------------------+------------------------------- >> Prof. Luigi RIZZO, ri...@iet.unipi.it <mailto:ri...@iet.unipi.it> . Dip. >> di Ing. dell'Informazione >> http://www.iet.unipi.it/~luigi/ <http://www.iet.unipi.it/~luigi/> . >> Universita` di Pisa >> TEL +39-050-2217533 . via Diotisalvi 2 >> Mobile +39-338-6809875 . 56122 PISA (Italy) >> -----------------------------------------+------------------------------- > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
