Hi Naveen,
See replies inline below...
On Fri, Aug 14, 2020 at 12:53 PM Naveen Joy (najoy) <na...@cisco.com> wrote:
> Thanks, Matthew. I am seeing the same behavior with the default
> advertisement interval of 1s.
>
> Tcpdump on a linux tap interface plugged into the same BD as the backup VR
> shows VRRP advertisements arriving at the configured rate of 1s (100cs),
>
> So, there is no packet loss of advertisements or delays in sending
> advertisements by the master VR.
>
>
>
> 10:37:19.991540 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:20.991619 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:21.991783 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:22.991792 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:23.991926 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:24.991976 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:25.992057 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:26.992131 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:27.992257 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:28.992311 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:29.992402 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:30.992513 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> 10:37:31.992586 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid
> 10, prio 110, intvl 100cs, length 12
>
> <truncated>
>
>
>
Ok, its good to know that the packets are all arriving on the backup.
The diagram attached to the first message in this thread said that the
master has address 10.4.4.1 and the backup has 10.4.4.3. Is that diagram
still accurate? The packets in the capture have a source address of
10.4.4.3, which is the address that is supposed to be configured on the
backup according to the diagram. If the diagram is still accurate, it seems
like those packets should be dropped by the backup as 'spoofed' packets
since their source address is configured locally on the backup.
If the diagram is not accurate and the master VR is truly supposed to be
advertising with a source address of 10.4.4.3, can you please use vppctl to
generate a packet capture on the backup VR? You could run something like
'vppctl trace add dpdk-input 50; sleep 10; vppctl show trace'. Depending on
how noisy your network is, that ought to capture a few inbound
advertisements.
> However, it appears that there is a delay in VRRP packet processing at the
> backup VR resulting in frequent state transitions.
>
>
>
> On the backup VR:
>
> vpp# show err
>
> Count Node Reason
>
> 120347 vrrp4-input VRRP packets processed
>
>
>
> vpp# show err (after 1 sec)
>
> Count Node Reason
>
> 120347 vrrp4-input VRRP packets processed
>
>
>
Is that the only output from 'show err' or did you clean up the output to
only include the counters which looked like they are related to VRRP? If
there are other counters displayed by 'show err' that you omitted from the
output, it would be helpful to see the full output.
> Also, log on the backup VR shows that VRRP advertisements from master are
> received every 4s
>
>
>
> Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0]
> sw_if_index 14 VR ID 10 IPv4 transitioning to Backup
>
> Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238:
> Deleting VR addresses on sw_if_index 14
>
> Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123:
> Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 13
>
> Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0]
> sw_if_index 14 VR ID 10 IPv4 transitioning to Master
>
> Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Adding
> VR addresses on sw_if_index 14
>
> Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Adding
> virtual MAC address 00:00:5e:00:01:0a on hardware interface 13
>
> Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0]
> sw_if_index 14 VR ID 10 IPv4 transitioning to Backup
>
> Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238:
> Deleting VR addresses on sw_if_index 14
>
> Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123:
> Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 13
>
> Aug 14 10:44:04 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0]
> sw_if_index 14 VR ID 10 IPv4 transitioning to Master
>
> Aug 14 10:44:04 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Adding
> VR addresses on sw_if_index 14
>
> Aug 14 10:44:04 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Adding
> virtual MAC address 00:00:5e:00:01:0a on hardware interface 13
>
> Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0]
> sw_if_index 14 VR ID 10 IPv4 transitioning to Backup
>
> Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238:
> Deleting VR addresses on sw_if_index 14
>
> Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123:
> Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 13
>
> Aug 14 10:44:08 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0]
> sw_if_index 14 VR ID 10 IPv4 transitioning to Master
>
> Aug 14 10:44:08 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Adding
> VR addresses on sw_if_index 14
>
> Aug 14 10:44:08 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Adding
> virtual MAC address 00:00:5e:00:01:0a on hardware interface 13
>
> Aug 14 10:44:09 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
>
>
>
>
Those log messages do not necessarily indicate that advertisements are only
being processed every 4s. When a VR is in backup state it does not log
anything about advertisements it receives unless there is something
significant about them. The messages above that are logged about
advertisements being received are only logged because the VR is in the
master state. The fact that it received an advertisement indicates that
either a peer is advertising when it should not be, or a higher priority VR
is preempting this one.
Thanks,
-Matt
>
>
> *From: *Matthew Smith <mgsm...@netgate.com>
> *Date: *Friday, August 14, 2020 at 6:14 AM
> *To: *"Naveen Joy (najoy)" <na...@cisco.com>
> *Cc: *"vpp-dev@lists.fd.io" <vpp-dev@lists.fd.io>
> *Subject: *Re: VRRP issue
>
>
>
> Hi Naveen,
>
>
>
> Generally a transition from backup to master occurs if the master down
> timer expires and no advertisement has been received. So it seems like some
> advertisement packets from the higher priority VR are not being received or
> are not being processed before the timer expires. Since the lower priority
> VR keeps switching from master to backup after receiving an advertisement
> from the higher priority VR, at least some of the advertisements from the
> higher priority VR are clearly being received.
>
>
>
> Packet loss of advertisements is the first thing that I suggest
> you investigate. A packet trace or capture on the lower priority VR would
> probably be helpful in determining whether the advertisements are arriving.
> It also might be helpful to see what 'vppctl show errors' says.
>
>
>
> Another possibility is that there are delays in sending advertisements by
> the higher priority VR or in receiving/processing them on the lower
> priority VR. The advertisement interval appears to be set to 0.3s and the
> master down timer is 1.08s. It seems unlikely that with packets being sent
> every 0.3s that either sending or receiving could be delayed by enough that
> the receiving side would not process one within 1.08s. Just to rule out
> that possibility, you could try increasing the advertisement interval to a
> higher value and see if the situation improves. Do you still see the same
> behavior if you configure the default advertisement interval of 1s (100cs)
> on both VRs?
>
>
>
> Thanks,
>
> -Matt
>
>
>
>
>
> On Thu, Aug 13, 2020 at 5:34 PM Naveen Joy (najoy) <na...@cisco.com>
> wrote:
>
> Hi Matthew/All,
>
>
>
> I am facing an issue with VRRP in VPP and would appreciate your help.
>
>
>
> (Attached - architecture diagram)
>
>
>
> 1. I have 2 nodes with VPP & in each node, VRRP is configured to back
> up a router BVI interface in a bridge domain.
> 2. The VRRP VRs are speaking VRRP (multicast) over an uplink VLAN
> interface connected to an external switch.
> 3. The active router has a VR priority of 110 and is set to preempt.
>
> The backup router has a VR priority of 100 and is not in preempt.
>
>
>
> 1. The issue is that VRRP in the backup router is unstable and keeps
> transitioning between the master and backup states every second.
>
> However, the VRRP in the master node is stable.
>
>
>
> I am running the latest VPP release installed from master this week.
>
>
>
> vpp# show version verbose
>
> Version: v20.09-rc0~283-g40c07ce7a~b1542
>
> Compiled by: root
>
> Compile host: 1f7cd9b19229
>
> Compile date: 2020-08-11T20:40:47
>
> Compile location: /w/workspace/vpp-merge-master-ubuntu1804
>
> Compiler: Clang/LLVM 9.0.0 (tags/RELEASE_900/final)
>
> Current PID: 5504
>
>
>
> *On the backup node –*
>
>
>
> Aug 13 12:59:48 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238:
> Deleting VR addresses on sw_if_index 11
>
> Aug 13 12:59:48 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123:
> Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10
>
> Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0]
> sw_if_index 11 VR ID 10 IPv4 transitioning to Master
>
> Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding
> VR addresses on sw_if_index 11
>
> Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding
> virtual MAC address 00:00:5e:00:01:0a on hardware interface 10
>
> Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4
>
> Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0]
> sw_if_index 11 VR ID 10 IPv4 transitioning to Backup
>
> Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238:
> Deleting VR addresses on sw_if_index 11
>
> Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123:
> Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10
>
> Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0]
> sw_if_index 11 VR ID 10 IPv4 transitioning to Master
>
> Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding
> VR addresses on sw_if_index 11
>
> Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding
> virtual MAC address 00:00:5e:00:01:0a on hardware interface 10
>
> Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4
>
> Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0]
> sw_if_index 11 VR ID 10 IPv4 transitioning to Backup
>
> Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238:
> Deleting VR addresses on sw_if_index 11
>
> Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123:
> Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10
>
> Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0]
> sw_if_index 11 VR ID 10 IPv4 transitioning to Master
>
> Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding
> VR addresses on sw_if_index 11
>
> Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding
> virtual MAC address 00:00:5e:00:01:0a on hardware interface 10
>
> Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4
>
> Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0]
> sw_if_index 11 VR ID 10 IPv4 transitioning to Backup
>
> Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238:
> Deleting VR addresses on sw_if_index 11
>
> Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123:
> Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10
>
>
>
> vpp# show vrrp vr
>
> [0] sw_if_index 11 VR ID 10 IPv4
>
> state Backup flags: preempt no accept yes unicast no
>
> priority: configured 100 adjusted 100
>
> timers: adv interval 30 master adv 30 skew 18 master down 108
>
> virtual MAC 00:00:5e:00:01:0a
>
> addresses 10.4.4.5
>
> peer addresses
>
> tracked interfaces
>
>
>
> vpp# show vrrp vr
>
> [0] sw_if_index 11 VR ID 10 IPv4
>
> state Master flags: preempt no accept yes unicast no
>
> priority: configured 100 adjusted 100
>
> timers: adv interval 30 master adv 30 skew 18 master down 108
>
> virtual MAC 00:00:5e:00:01:0a
>
> addresses 10.4.4.5
>
> peer addresses
>
> tracked interfaces
>
>
>
> *On the Master node ..*
>
>
>
> vpp# show vrrp vr
>
> [0] sw_if_index 14 VR ID 10 IPv4
>
> state Master flags: preempt yes accept yes unicast no
>
> priority: configured 110 adjusted 110
>
> timers: adv interval 30 master adv 30 skew 17 master down 107
>
> virtual MAC 00:00:5e:00:01:0a
>
> addresses 10.4.4.5
>
> peer addresses
>
> tracked interfaces
>
>
>
> Aug 13 13:29:44 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 13 13:29:45 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 13 13:29:46 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 13 13:29:47 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 13 13:29:49 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 13 13:29:50 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 13 13:29:51 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
> Aug 13 13:29:52 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received
> advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4
>
>
>
> VPP config:
>
>
>
> unix {
>
> nodaemon
>
> log /tmp/vpp.log
>
> full-coredump
>
> cli-listen /run/vpp/cli.sock
>
> gid vpp
>
> }
>
> api-trace {
>
> on
>
> }
>
> api-segment {
>
> uid najoy
>
> }
>
>
>
> Thanks,
>
> Naveen
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#17241): https://lists.fd.io/g/vpp-dev/message/17241
Mute This Topic: https://lists.fd.io/mt/76177843/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
