Hoi,
Quick followup, when I downgraded to 25.10-rc0~246-g49f6ec0c2 ARP
flooding stopped. Perhaps there is a regression between
25.10-rc0~246-g49f6ec0c2 and 25.10.0
The ARP flooding from the router below, that hurt users at LSIX, FrysIX
and SpeedIX today - https://paste.ipng.ch/vlur18.png
It's quite surprising how relatively little broadcast traffic (<1kqps)
can make router controlplanes unhappy.
groet,
Pim
On 28.11.2025 23:04, Pim van Pelt via lists.fd.io wrote:
Hoi,
I know this is a long shot, but this afternoon I upgraded one of my
routers at AS50869 from VPP 24.10 to VPP 25.10.0 (with the LinuxCP fix).
Shortly there-after, two internet exchanges (both with a /23
peeringlan) complained that the router was flooding ARP requests.
I could not see these in Linux CP, but they were visible in the
Internet Exchange when looking at a packet dump.
I could however see the ARP replies from the folks my router was
flooding, like so:
root@nlams0:/etc/bird/ebgp/groups# time tcpdump -evni speedix arp
16:08:46.638960 44:4c:a8:c7:4a:33 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.223.39 is-at 44:4c:a8:c7:4a:33, length 46
16:08:46.881844 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:46.886507 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:46.902967 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:46.905873 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:46.940812 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:46.973942 44:4c:a8:c7:4a:33 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.223.39 is-at 44:4c:a8:c7:4a:33, length 46
16:08:46.983844 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.003539 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.010575 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.030974 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.069999 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.123048 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.134061 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.158991 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
16:08:47.159000 c4:ca:2b:69:c8:f7 > b8:59:9f:e2:0a:9f, ethertype ARP
(0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply
185.1.222.21 is-at c4:ca:2b:69:c8:f7, length 46
In this network, the VPP router is b8:59:9f:e2:0a:9f and the router
that was being flooded is at c4:ca:2b:69:c8:f7. They saw the traffic
also. It immediately stopped when I set a static neighbor entry in
Linux. I took a trace from dpdk-input but it did not reveal any
outbound ARP traffic (which makes sense). It did however show the ARP
replies.
Could it be that between 24.10 and 25.10 release, something changed in
the ARP handling that might trigger an ARP flood from within arp
request/reply/ip-neighbor code? I'm hoping somebody can remember any
changes, I scanned over a bunch of changes but a year is a long time
and bisecting on an internet exchange is impractical. I may be able to
repro this behavior in a lab, but before I go deeper: does this ARP
flooding ring a bell for anybody ?
groet,
Pim
--
Pim van Pelt <[email protected]>
PBVP1-RIPE https://ipng.ch/
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#26583): https://lists.fd.io/g/vpp-dev/message/26583
Mute This Topic: https://lists.fd.io/mt/116519525/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
