W dniu 09.11.2018 o 17:21, David Ahern pisze:
On 11/9/18 3:20 AM, Paweł Staszewski wrote:
I just catch some weird behavior :)
All was working fine for about 20k packets
Then after xdp start to forward every 10 packets
Interesting. Any counter showing drops?
nothing that will fit
NIC statistics:
rx_packets: 187041
rx_bytes: 10600954
tx_packets: 40316
tx_bytes: 16526844
tx_tso_packets: 797
tx_tso_bytes: 3876084
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 38391
tx_nop: 2
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 187041
rx_csum_unnecessary: 0
rx_csum_none: 150011
rx_csum_complete: 37030
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 64893
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 2468
tx_csum_partial: 35955
tx_csum_partial_inner: 0
tx_queue_stopped: 0
tx_queue_dropped: 0
tx_xmit_more: 0
tx_recover: 0
tx_cqes: 38423
tx_queue_wake: 0
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 186302
rx_cache_full: 0
rx_cache_empty: 666768
rx_cache_busy: 174
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 249320
ch_poll: 249321
ch_arm: 249001
ch_aff_change: 0
ch_eq_rearm: 0
rx_out_of_buffer: 0
rx_if_down_packets: 57
rx_vport_unicast_packets: 142659
rx_vport_unicast_bytes: 42706914
tx_vport_unicast_packets: 40167
tx_vport_unicast_bytes: 16668096
rx_vport_multicast_packets: 39188170
rx_vport_multicast_bytes: 3466527450
tx_vport_multicast_packets: 58
tx_vport_multicast_bytes: 4556
rx_vport_broadcast_packets: 16343520
rx_vport_broadcast_bytes: 1031334602
tx_vport_broadcast_packets: 91
tx_vport_broadcast_bytes: 5460
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 40316
rx_packets_phy: 55674361
rx_crc_errors_phy: 0
tx_bytes_phy: 16839376
rx_bytes_phy: 4763267396
tx_multicast_phy: 58
tx_broadcast_phy: 91
rx_multicast_phy: 39188180
rx_broadcast_phy: 16343521
rx_in_range_len_errors_phy: 0
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 1
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 3792455
rx_65_to_127_bytes_phy: 51821620
rx_128_to_255_bytes_phy: 37669
rx_256_to_511_bytes_phy: 1481
rx_512_to_1023_bytes_phy: 434
rx_1024_to_1518_bytes_phy: 694
rx_1519_to_2047_bytes_phy: 20008
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 6
rx_err_lane_0_phy: 0
rx_err_lane_1_phy: 0
rx_err_lane_2_phy: 0
rx_err_lane_3_phy: 6
rx_buffer_passed_thres_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 82
outbound_pci_stalled_rd: 0
outbound_pci_stalled_wr: 0
outbound_pci_stalled_rd_events: 0
outbound_pci_stalled_wr_events: 0
rx_prio0_bytes: 4144920388
rx_prio0_packets: 48310037
tx_prio0_bytes: 16839376
tx_prio0_packets: 40316
rx_prio1_bytes: 481032
rx_prio1_packets: 7074
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 9074194
rx_prio2_packets: 106207
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 0
rx_prio3_packets: 0
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 0
rx_prio4_packets: 0
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 0
rx_prio5_packets: 0
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 371961810
rx_prio6_packets: 4006281
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 236830040
rx_prio7_packets: 3244761
tx_prio7_bytes: 0
tx_prio7_packets: 0
tx_pause_storm_warning_events : 0
tx_pause_storm_error_events: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
NIC statistics:
rx_packets: 843
rx_bytes: 58889
tx_packets: 324
tx_bytes: 23324
tx_tso_packets: 0
tx_tso_bytes: 0
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 293
tx_nop: 0
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 843
rx_csum_unnecessary: 0
rx_csum_none: 190
rx_csum_complete: 653
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 324
tx_csum_partial: 0
tx_csum_partial_inner: 0
tx_queue_stopped: 0
tx_queue_dropped: 0
tx_xmit_more: 1
tx_recover: 0
tx_cqes: 323
tx_queue_wake: 0
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 64926
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 47958
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 648
rx_cache_full: 0
rx_cache_empty: 602112
rx_cache_busy: 0
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 49628
ch_poll: 49628
ch_arm: 49626
ch_aff_change: 0
ch_eq_rearm: 0
rx_out_of_buffer: 0
rx_if_down_packets: 46
rx_vport_unicast_packets: 5953
rx_vport_unicast_bytes: 4927049
tx_vport_unicast_packets: 65194
tx_vport_unicast_bytes: 31820150
rx_vport_multicast_packets: 37085249
rx_vport_multicast_bytes: 2449620421
tx_vport_multicast_packets: 55
tx_vport_multicast_bytes: 4278
rx_vport_broadcast_packets: 434654
rx_vport_broadcast_bytes: 31881063
tx_vport_broadcast_packets: 1
tx_vport_broadcast_bytes: 60
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 65250
rx_packets_phy: 37525857
rx_crc_errors_phy: 0
tx_bytes_phy: 32085488
rx_bytes_phy: 2636532027
tx_multicast_phy: 55
tx_broadcast_phy: 1
rx_multicast_phy: 37085250
rx_broadcast_phy: 434654
rx_in_range_len_errors_phy: 0
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 0
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 63346
rx_65_to_127_bytes_phy: 37434768
rx_128_to_255_bytes_phy: 14088
rx_256_to_511_bytes_phy: 10461
rx_512_to_1023_bytes_phy: 96
rx_1024_to_1518_bytes_phy: 1933
rx_1519_to_2047_bytes_phy: 1165
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 5
rx_err_lane_0_phy: 1
rx_err_lane_1_phy: 0
rx_err_lane_2_phy: 0
rx_err_lane_3_phy: 4
rx_buffer_passed_thres_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 82
outbound_pci_stalled_rd: 0
outbound_pci_stalled_wr: 0
outbound_pci_stalled_rd_events: 0
outbound_pci_stalled_wr_events: 0
rx_prio0_bytes: 23157221
rx_prio0_packets: 195789
tx_prio0_bytes: 32085488
tx_prio0_packets: 65250
rx_prio1_bytes: 0
rx_prio1_packets: 0
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 0
rx_prio2_packets: 0
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 23397578
rx_prio3_packets: 343182
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 0
rx_prio4_packets: 0
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 0
rx_prio5_packets: 0
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 14643472
rx_prio6_packets: 203589
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 2575333474
rx_prio7_packets: 36783293
tx_prio7_bytes: 0
tx_prio7_packets: 0
tx_pause_storm_warning_events : 0
tx_pause_storm_error_events: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
But wondering if any offloading now can do some things that we dont want
for xdp
currently all offloads are enabled.
ethtool -k enp175s0f0
Features for enp175s0f0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
Also at the time when xdp is forwarding 1/10 frame - same problem is
with local input/output traffic - testing server is also responding to
1/10 icmp request
ping 172.16.0.2 -i 0.1
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.20 ms
64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=4.85 ms
64 bytes from 172.16.0.2: icmp_seq=29 ttl=64 time=4.91 ms
64 bytes from 172.16.0.2: icmp_seq=38 ttl=64 time=4.85 ms
64 bytes from 172.16.0.2: icmp_seq=48 ttl=64 time=5.00 ms
^C
--- 172.16.0.2 ping statistics ---
55 packets transmitted, 6 received, 89% packet loss, time 5655ms
rtt min/avg/max/mdev = 4.850/4.992/5.203/0.145 ms
And again after some time back to normal
ping 172.16.0.2 -i 0.1
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.02 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=5.06 ms
64 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=5.19 ms
64 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=5.07 ms
64 bytes from 172.16.0.2: icmp_seq=5 ttl=64 time=5.08 ms
64 bytes from 172.16.0.2: icmp_seq=6 ttl=64 time=5.14 ms
64 bytes from 172.16.0.2: icmp_seq=7 ttl=64 time=5.08 ms
64 bytes from 172.16.0.2: icmp_seq=8 ttl=64 time=5.17 ms
64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.04 ms
64 bytes from 172.16.0.2: icmp_seq=10 ttl=64 time=5.10 ms
64 bytes from 172.16.0.2: icmp_seq=11 ttl=64 time=5.11 ms
64 bytes from 172.16.0.2: icmp_seq=12 ttl=64 time=5.13 ms
64 bytes from 172.16.0.2: icmp_seq=13 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=14 ttl=64 time=5.15 ms
64 bytes from 172.16.0.2: icmp_seq=15 ttl=64 time=5.13 ms
64 bytes from 172.16.0.2: icmp_seq=16 ttl=64 time=5.04 ms
64 bytes from 172.16.0.2: icmp_seq=17 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=18 ttl=64 time=5.07 ms
64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=5.06 ms
64 bytes from 172.16.0.2: icmp_seq=20 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=21 ttl=64 time=5.21 ms
64 bytes from 172.16.0.2: icmp_seq=22 ttl=64 time=4.98 ms
^C
--- 172.16.0.2 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 2105ms
rtt min/avg/max/mdev = 4.988/5.104/5.210/0.089 ms
I will try to catch this with debug enabled
Wondering also - cause xdp will bypass now vlan counters and other stuff
like tcpdump
yes, xdp is before tcpdump based sockets.
And the counters (vlan just being the current example) is another
problem to be solved. The vlan net_device never sees the packet and you
can not arbitrarily bump the counters just because the device lookups
reference them.
Ok.
Is there possible to add only counters from xdp for vlans ?
This will help me in testing.
I will take a look today at adding counters that you can dump using
bpftool. It will be a temporary solution for this xdp program only.
Yes anything that can give me counters to check traffic lvls
And also - for non lab scenario there should be possible to sniff
sometimes on interface :)
Yes, sampling is another problem.
Soo wondering if need to attack another xdp program to interface or all
this can be done by one
I think this is time where i will need to learn more about xdp :)
