Public bug reported: Patch https://review.opendev.org/#/c/707406/ introduced new mechanism which brings all HA router's interfaces to be DOWN if router is in backup mode. And that works fine with keepalived 1.4.x but in keepalived 2.x.x it changed and keepalived now tracks by default interfaces of virtual_ipaddresses and routes. And will go to FAULT state if such interface is DOWN. That cause problem with router which will never be transitioned to master state.
We should add "no_track" option to qg- and qr- interfaces in keepalived config file now. Errors can be seen e.g. in result of tripleo job: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2b3/721574/1/check /tripleo-ci- centos-8-scenario007-standalone/2b3f794/logs/undercloud/var/log/journal.txt Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting Keepalived v2.0.10 (11/12,2018) Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Running on Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 (built for Linux 4.18.0) Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Command line: '/usr/sbin/keepalived' '-n' '-l' '-D' '-P' '-f' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf' '-p' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived' '-r' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived-vrrp' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '-D' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'. Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting VRRP child process, pid=147864 Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink reflector Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink command channel Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'. Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 22) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe0a:3675/64) - ignoring scope Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 23) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fea8:ae56/64) - ignoring scope Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) Ignoring track_interface ha-dcefeeaa-6d since own interface Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address 169.254.195.203 for interface ha-dcefeeaa-6d Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address fe80::f816:3eff:fec7:3b26 for interface ha-dcefeeaa-6d Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) entering FAULT state Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous ARP shared channel Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous NDISC shared channel Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing VIPs. Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing E-VIPs. Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes ** Affects: neutron Importance: Critical Assignee: Slawek Kaplonski (slaweq) Status: Confirmed ** Tags: l3-dvr-backlog -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1874211 Title: [L3HA] Keepalived 2.x.x tracks state of virtual_ipaddresses interfaces and router now Status in neutron: Confirmed Bug description: Patch https://review.opendev.org/#/c/707406/ introduced new mechanism which brings all HA router's interfaces to be DOWN if router is in backup mode. And that works fine with keepalived 1.4.x but in keepalived 2.x.x it changed and keepalived now tracks by default interfaces of virtual_ipaddresses and routes. And will go to FAULT state if such interface is DOWN. That cause problem with router which will never be transitioned to master state. We should add "no_track" option to qg- and qr- interfaces in keepalived config file now. Errors can be seen e.g. in result of tripleo job: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2b3/721574/1/check /tripleo-ci- centos-8-scenario007-standalone/2b3f794/logs/undercloud/var/log/journal.txt Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting Keepalived v2.0.10 (11/12,2018) Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Running on Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 (built for Linux 4.18.0) Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Command line: '/usr/sbin/keepalived' '-n' '-l' '-D' '-P' '-f' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf' '-p' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived' '-r' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived-vrrp' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: '-D' Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'. Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting VRRP child process, pid=147864 Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink reflector Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink command channel Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'. Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 22) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe0a:3675/64) - ignoring scope Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 23) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fea8:ae56/64) - ignoring scope Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) Ignoring track_interface ha-dcefeeaa-6d since own interface Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address 169.254.195.203 for interface ha-dcefeeaa-6d Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address fe80::f816:3eff:fec7:3b26 for interface ha-dcefeeaa-6d Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) entering FAULT state Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous ARP shared channel Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous NDISC shared channel Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing VIPs. Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing E-VIPs. Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1874211/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
