Hi,
I'm not sure how it behaved in earlier kernels (can check later), but it
is / looks bugged in at least recent 5.x+ ones (tests were done with
5.11.8 and 5.10.25).
Consider following setup:
# ip -o ad sh
1: lo inet 127.0.0.1/8 scope host lo
2: right1 inet 10.0.10.2/24 scope global
3: right2 inet 10.0.20.2/24 scope global
# ip ro sh tab main
default via 10.0.10.1 dev right1
10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2
# ip ro sh tab 123
default via 10.0.20.1 dev right2 src 10.0.20.2
And routing rules:
0: from all lookup local
9: from all fwmark 0x1 ipproto udp sport 1194 lookup 123
10: from all ipproto udp sport 1194 lookup 123
32766: from all lookup main
32767: from all lookup default
This - without any mangling via ipt/nft or by other means - works
correctly, for example:
nc -u -p 1194 1.2.3.4 12345
will be routed out correctly via 'right2' using 10.0.20.2
But if we add mark to locally outgoing packets:
iptables -t mangle -A OUTPUT -j MARK --set-mark 1
Then *both* rule 9 and rule 10 will be ignored during reroute check.
tcpdump on interface 'right1' will show:
# tcpdump -nvi right1 udp
tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot
length 262144 bytes
13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF],
proto UDP (17), length 33)
10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5
Initial routing decision in rule 10 will set the address correctly, but
the packet goes out via interface right1, ignoring both 9 and 10.
If I add another routing roule:
8: from all fwmark 0x1 lookup 123
Then the packects will flow correctly - but I *cannot* use (from the
ones I tested): sport, dport, ipproto, uidrange - as they will cause the
rule to be ignored. For example, this setup of routing rules will fail,
if there is any mark set on a packet (nc had uid 1120):
# ip ru sh
0: from all lookup local
10: from all ipproto udp lookup 123
10: from all sport 1194 lookup 123
10: from all dport 12345 lookup 123
10: from all uidrange 1120-1120 lookup 123
32766: from all lookup main
32767: from all lookup default
Adding correct fwmark to the above rules will have *no* effect either.
Only fwmark *alone* will work (or in combination with: iif, from, to -
from the ones I tested).
I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't
look like there is anything wrong there. I initially suspected lack of
'rule->mark &&' in mark related line - but considering that rules such
as 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look
like it's the culprit (and mark_mask covers that test either way).
OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the
culprit here - but I haven't analyzed them yet. Perhaps it's just an
issue of changing output interface incorrectly after ip_route_me_harder() ?
Is this a bug ? Or am I misinterpreting how 'reroute check' works after
initial routing decision ? One would expect routing rules during
post-mangle check to not be ignored out of the blue, only because packet
mark changed on the packet. Not mentioning both marks and routing rules
can be used for separate purposes (e.g. marks for shaping).
