I noticed that this happens when the router HA interface shows status: Down what can cause the ha interface to do down
Ajay From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Wednesday, November 29, 2017 at 4:14 PM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: [openstack-dev] [neutron] Neutron L3 agent/Keepalived Hi I have a case where after running the system for a while I see that floating ip association API gets accepted and no Errors in neutron l3 logs But when I go into the qrouter namespace and check the qg- interface the floating ip is not added Also the keepalived.conf is not updated and SIGUP of the keepalived process is not done So whats the place to look in this case. What is the flow in neutron l3 agent who adds the floating ip to the qg- interface I see the router update notification received Key log snippet attached. As you can see SIGUP of keepalived is missing and also I confirmed keepalived configs are not updated 2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: b3886b998c3049e799170bb5351300d1 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:269 2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent [req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff 13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification :[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed'] routers_updated /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413 2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None, priority 0 _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:487 2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568 2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416 2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [] for router, before_update _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142 2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process router updates process /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1105 2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105 2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150 2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282 2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105 2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150 2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212 2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105 2017-11-29 14:32:09.983 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150 2017-11-29 14:32:09.986 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-restore', '-n'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105 2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150 2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 7 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:576 2017-11-29 14:32:10.121 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225 2017-11-29 14:32:10.121 78 DEBUG neutron.agent.l3.router_info [-] Floating ip 5966c8b3-ff5c-4f01-91a6-f81d215bdeca added, status ACTIVE process_floating_ip_addresses /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:347 2017-11-29 14:32:10.122 78 DEBUG neutron.agent.l3.router_info [-] Sending floating ip statuses: {u'5966c8b3-ff5c-4f01-91a6-f81d215bdeca': 'ACTIVE'} update_fip_statuses /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:915 2017-11-29 14:32:10.122 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 18e6ceffec9549a5bd51f65551b65596 exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568 2017-11-29 14:32:10.150 78 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 18e6ceffec9549a5bd51f65551b65596 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416 2017-11-29 14:32:10.151 78 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212 2017-11-29 14:32:10.151 78 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105 2017-11-29 14:32:10.301 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150 2017-11-29 14:32:10.304 78 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 0 iptables commands were issued _apply_synchronized /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:576 2017-11-29 14:32:10.304 78 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225 2017-11-29 14:32:10.304 78 DEBUG neutron.callbacks.manager [-] Notify callbacks [('neutron.agent.linux.pd.update_router--9223372036851355335', <function update_router at 0x3431398>), ('neutron.agent.metadata.driver.after_router_updated-3681661', <function after_router_updated at 0x382d7d0>)] for router, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142 2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.update_router" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 2017-11-29 14:32:10.305 78 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.update_router" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282 2017-11-29 14:32:10.305 78 DEBUG neutron.agent.l3.agent [-] Finished a router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:536
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
