Hi Mathis, I'd recommend to test with a single worker 1st to rule out any side-effect related to multi-worker.
While you run iperf, can you do the following and share the output: ~# vppctl clear run && vppctl show run ~# vppctl clear err && vppctl show err ~# vppctl clear hard && vppctl show hard Also if you can share the output of ~# vppctl show log ~# vppctl show pci ~# vppctl show thread That would help understand your setup. Best ben ________________________________________ From: [email protected] <[email protected]> on behalf of mjoffre via lists.fd.io <[email protected]> Sent: Wednesday, December 3, 2025 17:16 To: [email protected] Subject: Re: [vpp-dev] Severe Latency on NAT64 (IPv6→IPv4) with VPP v25.10 #ipv6 #nat Hi Ole, Thanks for looking into this. I'm seeing repeated NAT64 and ACL-related errors, plus intermittent DPDK TX failures on TwentyFiveGigabitEthernet81/0/0. Full output: Count Node Reason Severity 7 acl-plugin-out-ip6-fa ACL permit packets error 7 acl-plugin-out-ip6-fa checked packets error 3 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error 1 VirtualEthernet0/0/3-output interface is down error 7621 nat64-out2in-handoff same worker error 5179 nat64-out2in-handoff do handoff error 6745 nat64-out2in no translation error 13 nat64-in2out-handoff do handoff error 65271 dpdk-input no error error 14 acl-plugin-out-ip6-fa ACL deny packets error 2950 acl-plugin-out-ip6-fa ACL permit packets error 2964 acl-plugin-out-ip6-fa checked packets error 73336 arp-reply ARP replies sent info 2 arp-reply ARP request IP4 source address lear info 10 ip6-glean neighbor solicitations sent info 2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error 15 ip6-icmp-input neighbor solicitations for unknown error 9 ip6-icmp-input neighbor advertisements sent info 2 ip6-icmp-input neighbor advertisements received info 920 nat64-out2in no translation error 2565 nat64-in2out-handoff do handoff error 24 acl-plugin-out-ip6-fa ACL permit packets error 24 acl-plugin-out-ip6-fa checked packets error 4 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error 2 TwentyFiveGigabitEthernet81/0/0-tx Tx packet drops (dpdk tx failure) error 765 nat64-out2in no translation error 2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error 461 nat64-out2in no translation error 2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error 331 TwentyFiveGigabitEthernet81/0/0-tx Tx packet drops (dpdk tx failure) error 624 nat64-out2in no translation error 981 nat64-in2out-handoff do handoff error 2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error 531 nat64-out2in no translation error 12 nat64-in2out-handoff do handoff error 2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error ________________________________ Here's my sessions + bib table NAT64 sessions: 2604:2dc0:400:dddd:0:2:0:20 51568 64:ff9b::369d:dd0 443 15.220.168.42 5832 54.157.13.208 443 protcol tcp vrf 3 2604:2dc0:400:dddd:0:2:0:20 51564 64:ff9b::369d:dd0 443 15.220.168.42 6903 54.157.13.208 443 protcol tcp vrf 3 2604:2dc0:400:dddd:0:2:0:20 51562 64:ff9b::369d:dd0 443 15.220.168.42 5757 54.157.13.208 443 protcol tcp vrf 3 2604:2dc0:400:dddd:0:2:0:22 42380 64:ff9b::8b3c:a058 5205 15.220.168.42 42400 139.60.160.88 5205 protcol tcp vrf 3 2604:2dc0:400:dddd:0:2:0:22 42366 64:ff9b::8b3c:a058 5205 15.220.168.42 39831 139.60.160.88 5205 protcol tcp vrf 3 NAT64 BIB entries: 2604:2dc0:400:dddd:0:2:0:20 51568 15.220.168.42 5832 protocol tcp vrf 3 dynamic 1 sessions 2604:2dc0:400:dddd:0:2:0:20 51564 15.220.168.42 6903 protocol tcp vrf 3 dynamic 1 sessions 2604:2dc0:400:dddd:0:2:0:20 51562 15.220.168.42 5757 protocol tcp vrf 3 dynamic 1 sessions 2604:2dc0:400:dddd:0:2:0:22 42380 15.220.168.42 42400 protocol tcp vrf 3 dynamic 1 sessions 2604:2dc0:400:dddd:0:2:0:22 42366 15.220.168.42 39831 protocol tcp vrf 3 dynamic 1 sessions ________________________________ I tried to capture an E2E flow, while I think both traces are not from the same request, so the delay between them might not properly transcribe the handoff duration. 17:41:50:085312: vhost-user-input VirtualEthernet0/0/2 queue 0 virtio flags: SINGLE_DESC Single descriptor packet virtio_net_hdr first_desc_len 130 flags 0x00 gso_type 0 num_buff 0 17:41:50:085316: ethernet-input frame: flags 0x1, hw-if-index 5, sw-if-index 5 IP6: 02:00:00:da:77:36 -> 02:00:00:da:77:36 17:41:50:085319: ip6-input ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808 tos 0x00, flow label 0xa0802, hop limit 64, payload length 64 ICMP echo_request checksum 0x3812 17:41:50:085321: ip6-lookup fib 2 dpo-idx 10 flow hash: 0x00000000 ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808 tos 0x00, flow label 0xa0802, hop limit 64, payload length 64 ICMP echo_request checksum 0x3812 17:41:50:085323: ip6-rewrite tx_sw_if_index 2 adj-idx 10 : ipv6 via 2001:db8::3:0:1 loop0: mtu:9000 next:4 flags:[] 02640000000302640000000386dd flow hash: 0x00000000 00000000: 02640000000302640000000386dd600a080200403a3f26042dc00400c0010000 00000020: 0002000000220064ff9b00000000000000000808080880003812196f0014275d 00000040: 306900000000e65c090000000000101112131415161718191a1b1c1d1e1f2021 00000060: 22232425262728292a2b2c2d2e2f3031323334353637559fd2fd1a9d 17:41:50:085324: loop0-output loop0 flags 0x00180005 IP6: 02:64:00:00:00:03 -> 02:64:00:00:00:03 ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808 tos 0x00, flow label 0xa0802, hop limit 63, payload length 64 ICMP echo_request checksum 0x3812 17:41:50:085327: ethernet-input IP6: 02:64:00:00:00:03 -> 02:64:00:00:00:03 17:41:50:085327: ip6-input ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808 tos 0x00, flow label 0xa0802, hop limit 63, payload length 64 ICMP echo_request checksum 0x3812 17:41:50:085328: ip6-sv-reassembly-feature [not fragmented or atomic fragment] 17:41:50:085329: nat64-in2out-handoff NAT64-IN2OUT-HANDOFF: next-worker 4 17:41:59:104234: handoff_trace HANDED-OFF: from thread 2 trace index 22 17:41:59:104234: nat64-in2out NAT64-in2out: sw_if_index 2, next index 0 17:41:59:114299: ip4-lookup fib 0 dpo-idx 7 flow hash: 0x00000000 ICMP: 15.220.168.42 -> 8.8.8.8 tos 0x00, ttl 63, length 84, checksum 0xb3a3 dscp CS0 ecn NON_ECN fragment id 0x0000 ICMP echo_request checksum 0x77a0 id 33816 17:41:59:114301: ip4-rewrite tx_sw_if_index 1 dpo-idx 7 : ipv4 via 15.220.168.46 TwentyFiveGigabitEthernet81/0/0: mtu:9000 next:3 flags:[] 001c7317c0235c2573ac596c0800 flow hash: 0x00000000 00000000: 001c7317c0235c2573ac596c080045000054000000003e01b4a30fcca82a0808 00000020: 0808080077a08418001d305d306900000000d3900900000000001011 17:41:59:114303: TwentyFiveGigabitEthernet81/0/0-output TwentyFiveGigabitEthernet81/0/0 flags 0x00180005 IP4: 5c:25:73:ac:59:6c -> 00:1c:73:17:c0:23 ICMP: 15.220.168.42 -> 8.8.8.8 tos 0x00, ttl 62, length 84, checksum 0xb4a3 dscp CS0 ecn NON_ECN fragment id 0x0000 ICMP echo_request checksum 0x77a0 id 33816 17:41:59:114304: TwentyFiveGigabitEthernet81/0/0-tx TwentyFiveGigabitEthernet81/0/0 tx queue 4 buffer 0xfda0f9: current data 20, length 98, buffer-pool 0, ref-count 1, trace handle 0x4000009 l2-hdr-offset 0 l3-hdr-offset 14 PKT MBUF: port 65535, nb_segs 1, pkt_len 98 buf_len 2176, data_len 98, ol_flags 0x0, data_off 148, phys_addr 0xbf683ec0 packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 IP4: 5c:25:73:ac:59:6c -> 00:1c:73:17:c0:23 ICMP: 15.220.168.42 -> 8.8.8.8 tos 0x00, ttl 62, length 84, checksum 0xb4a3 dscp CS0 ecn NON_ECN fragment id 0x0000 ICMP echo_request checksum 0x77a0 id 33816 ________________________________ Ping output shows high RTT: 64 bytes from 64:ff9b::808:808: icmp_seq=184 ttl=113 time=31.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=185 ttl=113 time=40.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=186 ttl=113 time=39.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=187 ttl=113 time=38.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=188 ttl=113 time=37.0 ms 64 bytes from 64:ff9b::808:808: icmp_seq=189 ttl=113 time=36.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=190 ttl=113 time=35.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=191 ttl=113 time=34.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=192 ttl=113 time=33.1 ms 64 bytes from 64:ff9b::808:808: icmp_seq=193 ttl=113 time=32.1 ms Mathis
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#26622): https://lists.fd.io/g/vpp-dev/message/26622 Mute This Topic: https://lists.fd.io/mt/116590423/21656 Mute #ipv6:https://lists.fd.io/g/vpp-dev/mutehashtag/ipv6 Mute #nat:https://lists.fd.io/g/vpp-dev/mutehashtag/nat Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
