W dniu 27.01.2018 o 23:23, Paweł Staszewski pisze:
Hi
Today I made some real life traffic tests with kernel 4.15.0-rc9
but when traffic reach 50Gbit/s and about 6Mpps cpou load rises fast
from 48% to 100% for all cpu cores.
Here is some graph that presenting how cpu load rises when there was
more pps.
https://ibb.co/mhD5ob
here is perf record from that time:
https://pastebin.com/3zqG1rvE
There is 8x 10G ixgbe 82599 interfaces teamed with teamd.
No traffic queueing - only pfifo fast on all interfaces.
No NAT or iptables forles other than INPUT (about 30rules)
All nic's have same ethtool settings:
ethtool -k eth0
Features for eth0:
Cannot get device udp-fragmentation-offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 2048
ethtool -c eth0
Coalesce parameters for eth0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
rx-usecs: 512
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
Peft top for kernel 4.15.0-rc9 below (all 40 cores 100% cpu load with
6.3Mpps)
20.96% [kernel] [k] queued_spin_lock_slowpath
5.51% [kernel] [k] ixgbe_poll
5.49% [kernel] [k] ixgbe_xmit_frame_ring
4.39% [kernel] [k] do_raw_spin_lock
4.29% [kernel] [k] sch_direct_xmit
4.11% [kernel] [k] fib_table_lookup
3.11% [team_mode_roundrobin] [k] rr_transmit
2.71% [kernel] [k] __dev_queue_xmit
2.62% [kernel] [k] __ptr_ring_peek
2.39% [kernel] [k] skb_release_data
2.18% [kernel] [k] dev_gro_receive
1.75% [kernel] [k] __qdisc_run
1.67% [kernel] [k] pfifo_fast_enqueue
1.57% [kernel] [k] netdev_pick_tx
1.56% [kernel] [k] page_frag_free
1.48% [kernel] [k] ip_finish_output2
1.38% [kernel] [k] __slab_free
1.36% [kernel] [k] skb_unref
1.34% [kernel] [k] ixgbe_maybe_stop_tx
1.30% [kernel] [k] vlan_do_receive
1.28% [kernel] [k] pfifo_fast_dequeue
1.23% [kernel] [k] virt_to_head_page
Same configuration kernel 4.15.0-rc3 (50% cpu load on all 40 cores with
6.3Mpps)
7.81% [kernel] [k] ixgbe_xmit_frame_ring
7.61% [kernel] [k] ixgbe_poll
7.09% [kernel] [k] do_raw_spin_lock
5.63% [kernel] [k] fib_table_lookup
5.19% [kernel] [k] __dev_queue_xmit
4.38% [team_mode_roundrobin] [k] rr_transmit
3.10% [kernel] [k] netdev_pick_tx
2.79% [kernel] [k] skb_release_data
2.34% [kernel] [k] dev_gro_receive
1.99% [kernel] [k] page_frag_free
1.96% [kernel] [k] skb_unref
1.92% [kernel] [k] virt_to_head_page
1.90% [kernel] [k] ixgbe_maybe_stop_tx
1.82% [kernel] [k] vlan_do_receive
1.74% [kernel] [k] ip_finish_output2
1.73% [kernel] [k] __build_skb
1.68% [kernel] [k] __slab_free
1.67% [kernel] [k] __netif_receive_skb_core
1.60% [kernel] [k] inet_gro_receive
1.49% [kernel] [k] netif_skb_features
1.35% [kernel] [k] ip_rcv
1.33% [kernel] [k] ipt_do_table
1.30% [kernel] [k] compound_head
1.26% [kernel] [k] dev_hard_start_xmit
1.18% [kernel] [k] put_page
1.13% [kernel] [k] tcp_gro_receive
1.13% [kernel] [k] ip_forward
0.99% [kernel] [k] validate_xmit_skb
0.97% [kernel] [k] ip_route_input_rcu
0.88% [kernel] [k] inet_lookup_ifaddr_rcu
0.81% [kernel] [k] pfifo_fast_dequeue
0.77% [kernel] [k] vlan_dev_hard_start_xmit
0.64% [kernel] [k] ___slab_alloc
