On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote: > Hi, > > I'm not sure how it behaved in earlier kernels (can check later), but it is > / looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and > 5.10.25). > > Consider following setup: > > # ip -o ad sh > 1: lo inet 127.0.0.1/8 scope host lo > 2: right1 inet 10.0.10.2/24 scope global > 3: right2 inet 10.0.20.2/24 scope global > > # ip ro sh tab main > default via 10.0.10.1 dev right1 > 10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2 > 10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2 > > # ip ro sh tab 123 > default via 10.0.20.1 dev right2 src 10.0.20.2 > > And routing rules: > > 0: from all lookup local > 9: from all fwmark 0x1 ipproto udp sport 1194 lookup 123 > 10: from all ipproto udp sport 1194 lookup 123 > 32766: from all lookup main > 32767: from all lookup default > > This - without any mangling via ipt/nft or by other means - works correctly, > for example: > > nc -u -p 1194 1.2.3.4 12345 > > will be routed out correctly via 'right2' using 10.0.20.2 > > But if we add mark to locally outgoing packets: > > iptables -t mangle -A OUTPUT -j MARK --set-mark 1 > > Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump > on interface 'right1' will show: > > # tcpdump -nvi right1 udp > tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length > 262144 bytes > 13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto > UDP (17), length 33) > 10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5 > > Initial routing decision in rule 10 will set the address correctly, but the > packet goes out via interface right1, ignoring both 9 and 10. > > If I add another routing roule: > > 8: from all fwmark 0x1 lookup 123 > > Then the packects will flow correctly - but I *cannot* use (from the ones I > tested): sport, dport, ipproto, uidrange - as they will cause the rule to be > ignored. For example, this setup of routing rules will fail, if there is any > mark set on a packet (nc had uid 1120): > > # ip ru sh > 0: from all lookup local > 10: from all ipproto udp lookup 123 > 10: from all sport 1194 lookup 123 > 10: from all dport 12345 lookup 123 > 10: from all uidrange 1120-1120 lookup 123 > 32766: from all lookup main > 32767: from all lookup default > > Adding correct fwmark to the above rules will have *no* effect either. Only > fwmark *alone* will work (or in combination with: iif, from, to - from the > ones I tested). > > I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look > like there is anything wrong there. I initially suspected lack of > 'rule->mark &&' in mark related line - but considering that rules such as > 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like > it's the culprit (and mark_mask covers that test either way). > > OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit > here - but I haven't analyzed them yet. Perhaps it's just an issue of > changing output interface incorrectly after ip_route_me_harder() ?
ip_route_me_harder() does not set source / destination port in the flow key, so it explains why fib rules that use them are not hit after mangling the packet. These keys were added in 4.17, but I don't think this use case every worked. You have a different experience? > > Is this a bug ? Or am I misinterpreting how 'reroute check' works after > initial routing decision ? One would expect routing rules during post-mangle > check to not be ignored out of the blue, only because packet mark changed on > the packet. Not mentioning both marks and routing rules can be used for > separate purposes (e.g. marks for shaping). >
