Hello,
I’m running into a rather complex issue where BIRD tries to reinsert
routes into the kernel routing table that are already in there. I’m
at the point of suspecting a netlink bug in the 5.x linux kernel.
However, I may be entirely off, and would like to hear what people with
more BIRD and/or netlink expertise think about it.
My setup is bird 2.0.7 on Ubuntu 20.04. I’ve been able to reproduce
this with the default 5.4.0-42-generic kernel as well as the ubuntu
mainline 5.7.0-050700-generic kernel. This bird instance has an IBGP
peering with another bird 2.0.7 instance, which sends an IPv4 full
table. The IBGP peering is set up over a GRE tunnel. There is no other
routing software running. I am seeing this issue on another server too,
ubuntu 20.04 with 5.4.0-42-generic. A third one with ubuntu 18.04 and a
4.15.0-20-generic kernel, is fine.
The problem presents as follows: a few thousand routes fail insertion
into the kernel routing table, marked by bird as ‘!’:
-------------------------
# birdc show route table master4|grep \!|wc -l
5838
-------------------------
The number varies, but is in the order of a few thousand.
However, all routes seem to be in the kernel table:
-------------------------
# birdc show route count
BIRD 2.0.7 ready.
788748 of 788748 routes for 788747 networks in table master4
0 of 0 routes for 0 networks in table master6
Total: 788748 of 788748 routes for 788747 networks in 2 tables
# ip route|grep bird|wc -l
788748
-------------------------
When I wait a few minutes and check again, the set of failed prefixes
is almost entirely different. Apparently, the previously failed prefixes
have been “fixed”, but now the same problem appears with different ones.
When I check a particular sample, the route really is in the kernel
routing table already:
-------------------------
# birdc show route 81.200.176.0/20
BIRD 2.0.7 ready.
Table master4:
81.200.176.0/20 unicast [fra1 14:47:20.892] ! (100) [AS50664i]
via 10.195.6.1 on tun-fra1
# ip route|grep 81.200.176.0/20
81.200.176.0/20 via 10.195.6.1 dev tun-fra1 proto bird metric 32
# birdc show route 81.200.176.0/20
BIRD 2.0.7 ready.
Table master4:
81.200.176.0/20 unicast [fra1 14:47:20.892] ! (100) [AS50664i]
via 10.195.6.1 on tun-fra1
-------------------------
The logs are repeats of:
-------------------------
Aug 01 09:50:23 fra2 bird[83586]: Netlink: File exists
Aug 01 09:50:23 fra2 bird[83586]: Netlink: File exists
Aug 01 09:50:23 fra2 bird[83586]: Netlink: File exists
Aug 01 09:50:23 fra2 bird[83586]: Netlink: File exists
Aug 01 09:50:23 fra2 bird[83586]: Netlink: File exists
Aug 01 09:50:23 fra2 bird[83586]: ...
Aug 01 09:50:23 fra2 bird[83586]: I/O loop cycle took 5110 ms for 1 events
-------------------------
Seemingly, bird thinks routes are not yet in the kernel routing table,
tries to insert them, which fails because they are already there.
Later scans move the problem to different prefixes.
If I remove the tunneled IBGP peering, and set up an upstream peering
on a directly connected non-tunneled peer, everything works fine. If I
enable both the tunneled IBGP peering and the non-tunneled upstream,
the issue does appear. The routes that are affected by the failed
insertion attempts are then from both peers.
The prefixes that are affected are almost entirely cases where multiple
prefixes exist in the DFZ with the same network address, but different
lengths. Above I mentioned seeing 81.200.176.0/20 at some point, and
81.200.176.0/24 is also in the DFZ. From two samples of failed prefixes
at different points in time, about 98% are prefixes of this kind, where
the DFZ contains the same network address with a different length too.
This distribution is not reflected in the whole v4 full table,
suggesting that routes that meet this case, are most likely to fail
insertion.
I have done some deeper digging with strace.
https://p.6core.net/p/RtQ7xSSYd560i7FioG78ebsi is an strace of BIRD
starting and loading the v4 full table from the IBGP peer. I have
filtered this for “45.189.104.0” “GETROUTE” to trim down. I have the
full strace, but it’s 8GB:
1. Initially, bird does GETROUTE a few times, before the BGP session is
established.
2. Near the start, at 09:06:33.208648 and 09:06:33.209003, the BGP
session seems established, and 45.189.104.0/24 and 45.189.104.0/22
are inserted correctly. (Acknowledgement not in paste as it does not
contain the prefix.)
3. At 09:06:39.774454 bird sends to birdc that 45.189.104.0/24 is best
route and inserted correctly.
4. At 09:06:49.915838, bird sends a GETROUTE. Only the reply lines that
contain “45.189.104.0” are in the paste. Apparently, only
45.189.104.0/24 is included in the reply. 45.189.104.0/22 does not
seem to appear in the reply to GETROUTE.
5. At 09:07:08.455767, bird tries to insert 45.189.104.0/22 into the
kernel routing table. This fails, because it already exists.
6. At 09:07:43.144207 bird replies to birdc that 45.189.104.0/22 failed
to insert.
7. At 09:07:49.874282, bird does another GETROUTE. Both the /22 and the
/24 are included in the response.
8. At 09:08:25.623269, bird tries to update 45.189.104.0/22 which
succeeds. Note the different flags from 09:07:08.455767, specifically
NLM_F_EXCL vs NLM_F_REPLACE, presumably because bird is aware there
is an existing route.
9. Future GETROUTEs return both the /24 and /22, bird does nothing.
Working theory: netlink GETROUTE in at least some 5.x kernels may not
return all routes, when at least some routes in the table have a next
hop that is a tunnel interface, and this is almost entirely contained
to cases where multiple prefixes exist with the same network address.
Thoughts? And if that theory is correct, can we work around it in BIRD?
A few things I’ve changed that had no effect:
- Changing the tunnel type.
- Changing the peering from IBGP to EBGP.
- Using IPv6 addresses for the tunnel endpoints, and removing the static
route in the config below.
This is the full config I’m currently using:
-------------------------
log syslog all;
router id 141.98.136.36;
protocol kernel {
scan time 20;
ipv4 {
export all;
import none;
};
}
# never route the tunnel through the tunnel
protocol static tunnel_endpoints {
ipv4;
route 45.12.69.14/32 via 141.98.136.33;
}
protocol device {
}
# ibgp peering over GRE tunnel, ipv4 full table
protocol bgp fra1 {
local as 213279;
source address 10.195.6.2;
neighbor 10.195.6.1 as 213279;
direct;
ipv4 {
import all;
export none;
};
}
-------------------------
Sasha
