On 4/12/18 10:54 AM, Jeff Barnhill wrote:
> Hi David,
>
> In the slides referenced, you recommend adding an "unreachable
> default" route to the end of each VRF route table. In my testing (for
> v4) this results in a change to fib lookup failures such that instead
> of ENETUNREACH being returned, EHOSTUNREACH is returned since the fib
> finds the unreachable route, versus failing to find a route
> altogether.
>
> Have the implications of this been considered? I don't see a
> clean/easy way to achieve the old behavior without affecting non-VRF
> routing (eg. remove the unreachable route and delete the non-VRF
> rules). I'm guessing that programmatically, it may not make much
> difference, ie. lookup fails, but for debugging or to a user looking
> at it, the difference matters. Do you (or anyone else) have any
> thoughts on this?
We have recommended moving the local table down in the FIB rules:
# ip ru ls
1000: from all lookup [l3mdev-table]
32765: from all lookup local
32766: from all lookup main
32767: from all lookup default
and adding a default route to VRF tables:
# ip ro ls vrf red
unreachable default metric 4278198272
172.16.2.0/24 proto bgp metric 20
nexthop via 169.254.0.1 dev swp3 weight 1 onlink
nexthop via 169.254.0.1 dev swp4 weight 1 onlink
# ip -6 ro ls vrf red
2001:db8:2::/64 proto bgp metric 20
nexthop via fe80::202:ff:fe00:e dev swp3 weight 1
nexthop via fe80::202:ff:fe00:f dev swp4 weight 1
anycast fe80:: dev lo proto kernel metric 0 pref medium
anycast fe80:: dev lo proto kernel metric 0 pref medium
fe80::/64 dev swp3 proto kernel metric 256 pref medium
fe80::/64 dev swp4 proto kernel metric 256 pref medium
ff00::/8 dev swp3 metric 256 pref medium
ff00::/8 dev swp4 metric 256 pref medium
unreachable default dev lo metric 4278198272 error -101 pref medium
Over the last 2 years we have not seen any negative side effects to this
and is what you want to have proper VRF separation.
Without a default route lookups will proceed to the next fib rule which
means a lookup in the next table and barring other PBR rules will be the
main table. It will lead to wrong lookups.
Here is an example:
ip netns add foo
ip netns exec foo bash
ip li set lo up
ip li add red type vrf table 123
ip li set red up
ip li add dummy1 type dummy
ip addr add 10.100.1.1/24 dev dummy1
ip li set dummy1 master red
ip li set dummy1 up
ip li add dummy2 type dummy
ip addr add 10.100.1.1/24 dev dummy2
ip li set dummy2 up
ip ro get 10.100.2.2
ip ro get 10.100.2.2 vrf red
# ip ru ls
0: from all lookup local
1000: from all lookup [l3mdev-table]
32766: from all lookup main
32767: from all lookup default
# ip ro ls
10.100.1.0/24 dev dummy2 proto kernel scope link src 10.100.1.1
10.100.2.0/24 via 10.100.1.2 dev dummy2
# ip ro ls vrf red
10.100.1.0/24 dev dummy1 proto kernel scope link src 10.100.1.1
That's the setup. What happens on route lookups?
# ip ro get vrf red 10.100.2.1
10.100.2.1 via 10.100.1.2 dev dummy2 src 10.100.1.1 uid 0
cache
which is clearly wrong. Let's look at the lookup sequence
# perf record -e fib:* ip ro get vrf red 10.100.2.1
10.100.2.1 via 10.100.1.2 dev dummy2 src 10.100.1.1 uid 0
cache
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.003 MB perf.data (4 samples) ]
# perf script --fields trace:trace
table 255 oif 2 iif 1 src 0.0.0.0 dst 10.100.2.1 tos 0 scope 0 flags 4
table 123 oif 2 iif 1 src 0.0.0.0 dst 10.100.2.1 tos 0 scope 0 flags 4
table 254 oif 2 iif 1 src 0.0.0.0 dst 10.100.2.1 tos 0 scope 0 flags 4
nexthop dev dummy2 oif 4 src 10.100.1.1
The first one is because I did not move the local table down.
The second one is the correct vrf lookup
The third one is the continuation to the next table - the main table.
Adding a default route:
# ip ro add vrf red unreachable default
And the lookup is proper:
# ip ro get vrf red 10.100.2.1
RTNETLINK answers: No route to host
