Hi Neale, I will attempt this on this latest.
Yes ARP cache flush is one option as that seems to help out , I find that arp flush per interface requires a walk through all the arp entries and identify those that are specifically on the interface in question. Not sure whether I am missing something better. Thanks a lot for your help and valuable feedback. Thanks Rupesh On Thu, Feb 4, 2021 at 6:01 PM Neale Ranns <ne...@graphiant.com> wrote: > > > Hi Rupesh, > > > > 19.08 is no longer supported, please upgrade. > > > > The down side of flushing the ARP cache on link down is that the ARP > cached gets flushed 😉 you have to rebuild it when the link comes back > up, this takes time. Otherwise I see no issue with doing so. > > > > The adj walk is there to remove that adj/path from the ECMP set when the > interface goes down, it does not conflict with flushing the ARP cache, > which will make the adj incomplete. > > > > /neale > > > > > > *From: *Rupesh Raghuvaran <rupesh.raghuva...@gmail.com> > *Date: *Thursday, 4 February 2021 at 12:58 > *To: *Neale Ranns <ne...@graphiant.com> > *Cc: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *Subject: *Re: [vpp-dev] Fib entries as per show ip fib for prefix has > forwarding UNRESOLVED though packet is forwarded. > > Hi Neale, > > > > This is version 19.08. > > > > Meanwhile, based on whatever I understood from the discussion so far, I > experimented by adding a link down function that does an arp flush on the > respective interface which resulted in arp delete and subsequent adj nbr > removal. This seems to provide the desired result of the fib entry getting > marked resolved. Note currently there is only arp flush done on the admin > state change in ethernet/arp.c. Do you see any issues or potential downside > with this arp flush being done on link down ? > > There is also a backwalk associated with interface link up/down > via adj_nbr_interface_state_change_one, I am not sure about what is > expected from that walk, and the above handling from arp flush have any > conflicts ? > > > > Thanks > > Rupesh > > > > On Thu, Feb 4, 2021 at 3:50 PM Neale Ranns <ne...@graphiant.com> wrote: > > > > What VPP version is this? > > > > /neale > > > > *From: *Rupesh Raghuvaran <rupesh.raghuva...@gmail.com> > *Date: *Wednesday, 3 February 2021 at 17:39 > *To: *Neale Ranns <ne...@graphiant.com> > *Cc: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *Subject: *Re: [vpp-dev] Fib entries as per show ip fib for prefix has > forwarding UNRESOLVED though packet is forwarded. > > Hi Neale, > > > > Looking at the show ip arp we see that the arp entries still remain the > same even after the link is down. > > > > show ip arp > Time IP4 Flags Ethernet Interface > 1.9954 10.0.0.14 D de:ad:de:ad:00:04 GigabitEthernet0/3/0 > 1.9633 10.0.0.15 D de:ad:de:ad:00:05 GigabitEthernet0/4/0 > 2.8591 10.0.1.132 S 00:65:0d:98:00:00 loop10 > 2.8611 10.0.2.132 S 00:65:0e:bf:00:00 loop10 > 2.8626 12.0.1.2 S 00:65:0c:ca:00:00 loop10 > Proxy arps enabled for: > Fib_index 0 0.0.0.0 - 255.255.255.255 > > ---- > > show hardware GigabitEthernet0/4/0 > Name Idx Link Hardware > GigabitEthernet0/4/0 3 down GigabitEthernet0/4/0 > Link speed: 10 Gbps > Ethernet address de:ad:de:ad:00:01 > Red Hat Virtio > carrier down > flags: admin-up pmd maybe-multiseg > > <sniped> > ---- > > show hardware GigabitEthernet0/3/0 > Name Idx Link Hardware > GigabitEthernet0/3/0 2 up GigabitEthernet0/3/0 > Link speed: 10 Gbps > Ethernet address de:ad:de:ad:00:01 > Red Hat Virtio > carrier up full duplex mtu 9206 > flags: admin-up pmd maybe-multiseg > rx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1) > tx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1) > <snipped> > > > > Regarding the ARP responses dropped my understanding is that for an ARP > reply are acceptable if we have a fib source via a api/cli on the interface > enabling a successful arp reply src ip address lookup. That is essentially > saying that the such a specific arp reply is valid as per the > configuration, i am not sure about any harm with such a configuration as it > is correct as per topology. > > > > Thanks > > Rupesh > > > > On Wed, Feb 3, 2021 at 7:15 PM Neale Ranns <ne...@graphiant.com> wrote: > > > > Hi Rupesh, > > > > Dropping those ARP responses is a clue that you’re not doing something > right 😉 > > > > I would expect the ARP entry, adj and adj-source on the fib entry to be > removed (in that order) when the link goes down. ‘sh ip eighbours’ please. > > > > > https://github.com/FDio/vpp/blob/master/docs/gettingstarted/developers/fib20/routes.rst#adjacency-source-fib-entries > > > > /neale > > > > > > *From: *Rupesh Raghuvaran <rupesh.raghuva...@gmail.com> > *Date: *Wednesday, 3 February 2021 at 12:03 > *To: *Neale Ranns <ne...@graphiant.com> > *Cc: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *Subject: *Re: [vpp-dev] Fib entries as per show ip fib for prefix has > forwarding UNRESOLVED though packet is forwarded. > > Hi Neale, > > > > Thanks for your reply. > > > > I am adding a default route via 10.0.0.15 dev Ge0/4/0, a subnet route > 10.0.0.0/24 via 10.0.0.15 dev Ge0/4/0 and a host route to the > 10.0.0.15/32 on dev Ge0/4/0. No explicit arp entry is added. Without the > specific 10.0.0.15/32 on dev Ge0/4/0 link the arp ipv4 response for > 10.0.0.15 gets dropped for the reason src subnet not local to interface. > > On the link down of Ge0/4/0 all the routes specified above gets deleted. > But on the other link Ge0/3/0 we have similar routes, default route and > subnet route via 10.0.0.14 and host route to 10.0.0.14/32 on dev > Ge0/3/0. > > > > As per the fib log the src API based path gets removed but the src adj > based one remains. Even if the interface is down the adjacency source entry > remains the cover seems to get updated. to 7 , which is on a different > interface ie Ge0/3/0 than that is in the adjacency is still on Ge0/4/0. > > > > show ip fib 10.0.0.15 > ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] > locks:[src:plugin-hi:2, src:DHCP:7, src:adjacency:5, src:default-route:1, ] > 10.0.0.15/32 fib:0 index:8 locks:2 > src:adjacency refs:1 entry-flags:attached, > src-flags:added,contributing,active, cover:7 > path-list:[17] locks:2 uPRF-list:7 len:1 itfs:[3, ] > path:[18] pl-index:17 ip4 weight=1 pref=0 attached-nexthop: > 10.0.0.15 GigabitEthernet0/4/0^M > [@0]: ipv4 via 10.0.0.15 GigabitEthernet0/4/0: mtu:9000 > deaddead0005deaddead00010800 > Extensions: > path:18 > forwarding: UNRESOLVED > > > > and cover fib entry 7 specified above is the following. > > 7@10.0.0.0/24 > unicast-ip4-chain > [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:36 > to:[1770:254632]] > [0] [@5]: ipv4 via 10.0.0.14 GigabitEthernet0/3/0: mtu:9000 > deaddead0004deaddead00010800 > > > > > > When the interface is down is the corresponding fib entry sourced by the > adjacency expected to be deleted. ? Could you also explain a bit on the > adjacency source refinement? > > > > Thanks > > Rupesh > > > > > > > > > > On Wed, Feb 3, 2021 at 2:40 PM Neale Ranns <ne...@graphiant.com> wrote: > > Hi Rupesh, > > > > 10.0.0.15 remains unresolved after link down because there remains an > adjacency/ARP-entry for it on Ge0/4/0 – did you add a static one? It is > unresolved because it fails the adjacency source refinement criteria. > Packets to 10.0.0.15 are forwarded using the default route. This is > expected behaviour. > > > > Your use of unnumbered is unconventional. You’d get a better experience if > you used standard IP addressing. For example, add separate /31s on your > gigEs, then another /32 on the loopbacks. Add routes to the peer’s > loopbacks via your control plane agent. The tunnel endpoints should be the > loopback addresses. > > > > /neale > > > > > > *From: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> on behalf of Rupesh > Raghuvaran via lists.fd.io <rupesh.raghuvaran=gmail....@lists.fd.io> > *Date: *Tuesday, 2 February 2021 at 17:08 > *To: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *Subject: *[vpp-dev] Fib entries as per show ip fib for prefix has > forwarding UNRESOLVED though packet is forwarded. > > > > VPP is configured as router (R0) connects to two routers referred as R1 > and R2 , and there is direct link between R1 and R2 is connected. > > > > R0 has a loopback interface loop0 configured with 10.0.0.16/32 and > 10.0.0.18/32 , interface Ge0/3/0 and Ge0/4/0 is unnumbered to loop0 > > R0 Ge0/3/0 connects to R1 > > R0 Ge0/4/0 connecte to R2 > > > > R1 has a loopback interface with 10.0.0.14/32 and the R1-R0 interface > is unnumbered to that interface. > > R2 has a loopback interface with 10.0.0.15/32 and the R2-R0 interface > is unnumbered to that interface. > > > > > > R0 is configured with following routes > > > > 0.0.0.0/0 via 10.0.0.14 > > 0.0.0.0/0 via 10.0.0.15 > > 10.0.0.0/24 via 10.0.0.14 > > 10.0.0.0/24 via 10.0.0.15 > > 10.0.0.14/32 via Ge0/3/0 > > 10.0.0.15/32 via Ge0/4/0 > > > > with this configuration I am able to ping 10.0.0.15 and 10.0.0.14 on > respective link. > > > > show ip fib 10.0.0.15 > > ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] > locks:[src:plugin-hi:2, src:DHCP:7, src:adjacency:5, src:default-route:1, ] > > 10.0.0.15/32 fib:0 index:8 locks:3 > > src:API refs:1 entry-flags:attached, src-flags:added,contributing,active, > > path-list:[12] locks:2 flags:shared, uPRF-list:14 len:1 itfs:[3, ] > > path:[12] pl-index:12 ip4 weight=1 pref=0 attached-nexthop: > oper-flags:resolved, cfg-flags:attached, > > 10.0.0.15 GigabitEthernet0/4/0 > > [@0]: ipv4 via 10.0.0.15 GigabitEthernet0/4/0: mtu:9000 > deaddead0005deaddead00010800 > > > > src:adjacency refs:1 entry-flags:attached, src-flags:added, cover:-1 > > path-list:[17] locks:1 uPRF-list:7 len:1 itfs:[3, ] > > path:[18] pl-index:17 ip4 weight=1 pref=0 attached-nexthop: > oper-flags:resolved, > > 10.0.0.15 GigabitEthernet0/4/0^M > > [@0]: ipv4 via 10.0.0.15 GigabitEthernet0/4/0: mtu:9000 > deaddead0005deaddead00010800 > > Extensions: > > path:18 > > forwarding: unicast-ip4-chain > > [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:14 > to:[2020:207099]] > > [0] [@5]: ipv4 via 10.0.0.15 GigabitEthernet0/4/0: mtu:9000 > deaddead0005deaddead00010800 > > > > > > On Link down of Ge0/4/0 the control plange agent on the link down event > removes following routes > > ie corresponding to the one via the specific link set to down. > > 0.0.0.0/0 via 10.0.0.15 > > 10.0.0.0/24 via 10.0.0.15 > > 10.0.0.15/32 via Ge0/4/0 > > > > When the Ge0/4/0 link is down fib entry for 10.0.0.15 has forwarding > marked as UNRESOLVED. > > > > show ip fib 10.0.0.15 > > ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] > locks:[src:plugin-hi:2, src:DHCP:7, src:adjacency:5, src:default-route:1, ] > > 10.0.0.15/32 fib:0 index:8 locks:2 > > src:adjacency refs:1 entry-flags:attached, > src-flags:added,contributing,active, cover:7 > > path-list:[17] locks:2 uPRF-list:7 len:1 itfs:[3, ] > > path:[18] pl-index:17 ip4 weight=1 pref=0 attached-nexthop: > > 10.0.0.15 GigabitEthernet0/4/0 > > [@0]: ipv4 via 10.0.0.15 GigabitEthernet0/4/0: mtu:9000 > deaddead0005deaddead00010800 > > Extensions: > > path:18 > > forwarding: UNRESOLVED > > > > Its found that the packets destined to 10.0.0.15 is send out via the > Ge0/3/0 as expected even though it is marked unresolved. > > ie ping from R0 to R2 ip 10.0.0.15 via R1 when link from R0 to R2 is down > > > > > > 00:13:42:303487: ip4-input > > ICMP: 10.0.0.16 -> 10.0.0.15 > > tos 0x00, ttl 64, length 84, checksum 0xbd3c > > fragment id 0x694e, flags DONT_FRAGMENT > > ICMP echo_request checksum 0xe43a > > 00:13:42:303492: ip4-lookup > > fib 0 dpo-idx 3 flow hash: 0x00000000 > > ICMP: 10.0.0.16 -> 10.0.0.15 > > tos 0x00, ttl 64, length 84, checksum 0xbd3c > > fragment id 0x694e, flags DONT_FRAGMENT > > ICMP echo_request checksum 0xe43a > > 00:13:42:303497: ip4-rewrite > > tx_sw_if_index 2 dpo-idx 3 : ipv4 via 10.0.0.14 GigabitEthernet0/3/0: > mtu:9000 deaddead0004deaddead00010800 flow hash: 0x00000000 > > 00000000: > deaddead0004deaddead0001080045000054694e40003f01be3c0a0000100a00 > > 00000020: 000f0800e43a0a4700504066196000000000ea940600000000001011 > > 00:13:42:303500: GigabitEthernet0/3/0-output > > GigabitEthernet0/3/0 > > IP4: de:ad:de:ad:00:01 -> de:ad:de:ad:00:04 > > ICMP: 10.0.0.16 -> 10.0.0.15 > > tos 0x00, ttl 63, length 84, checksum 0xbe3c > > fragment id 0x694e, flags DONT_FRAGMENT > > ICMP echo_request checksum 0xe43a > > 00:13:42:303503: GigabitEthernet0/3/0-tx > > GigabitEthernet0/3/0 tx queue 0 > > buffer 0x18e07: current data -14, length 98, buffer-pool 0, ref-count 1, > trace handle 0x28 > > PKT MBUF: port 65535, nb_segs 1, pkt_len 98 > > buf_len 2176, data_len 98, ol_flags 0x0, data_off 114, phys_addr > 0x73a38240 > > packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > > IP4: de:ad:de:ad:00:01 -> de:ad:de:ad:00:04 > > ICMP: 10.0.0.16 -> 10.0.0.15 > > tos 0x00, ttl 63, length 84, checksum 0xbe3c > > fragment id 0x694e, flags DONT_FRAGMENT > > ICMP echo_request checksum 0xe43a > > > > But for any gre tunnels with l2 adj index which refers to 10.0.0.15 seems > to have forwarding dpo-drop as drop. > > Why is the fib entry for 10.0.0.15 remains with forwarding unresolved ? > > I hope the configuration attempted is supported. Please find the related > logs and various fib related cli outputs before and after. > > Thanks in Advance > Rupesh > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18677): https://lists.fd.io/g/vpp-dev/message/18677 Mute This Topic: https://lists.fd.io/mt/80318226/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
