Reviewed: https://review.openstack.org/429923 Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=6bf84e7afbf07ab33907150d59d0b33d053240b6 Submitter: Jenkins Branch: master
commit 6bf84e7afbf07ab33907150d59d0b33d053240b6 Author: Cedric Brandily <[email protected]> Date: Tue Feb 7 00:16:08 2017 +0100 Do not complain in firewall_group_deleted if the FW is already deleted Currently firewall_group_deleted[1] crashs if the firewall is already deleted or deleted concurrently during firewall_deleted call. We should avoid such behavior as there is no reason to crash if someone already did the job for us (ie: delete the FW). Moreover such crash is costly because it triggers a service-sync on FWaaS l3-reference agent (at least). Typically on a L3-DVR deployment, all firewall_deleted calls except the first one will fail so quite every L3-DVR will perform a FWaaS service-sync. This change updates firewall_group_deleted in order to succeed if the firewall is already deleted or if the firewall is deleted concurrently. [1] neutron.services.firewall.fwaas_plugin_v2.FirewallCallbacks Change-Id: Ic0b228896c8129205224417506bb06471e432955 Closes-Bug: #1658060 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1658060 Title: FirewallNotFound exceptions when deleting the firewall in FWaaS-DVR Status in neutron: Fix Released Bug description: We have four nodes, and we deploy both the FWaaS and DVR services. When deleting the firewall, we always get three FirewallNotFound exceptions. At present, we believe that, in DVR environment, evey node would run a L3-agent service. This causes a plugin corresponding to multiple agents. And each agent will call back the plugin's firewall_deleted() (neutron_fwaas/services/firewall/fwaas_plugin.py) to delete the instance in DB, but only the first agent will succeed. How to reproduce: - first create a firewall applied to a DVR router - then delete it $ neutron router-show test-fwaas +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | distributed | True | | external_gateway_info | | | ha | False | | id | cfa3e65e-d101-4cc7-80e5-39daf72c6572 | | name | test-fwaas | | routes | | | status | ACTIVE | | tenant_id | fc170b1b8a9a467b9e1a63d85ced5a86 | +-----------------------+--------------------------------------+ $ neutron firewall-create --name fw --router test-fwaas policy Created a new firewall: +--------------------+--------------------------------------+ | Field | Value | +--------------------+--------------------------------------+ | admin_state_up | True | | description | | | firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 | | id | afd38a9e-cf0a-4667-94e0-853a888fd981 | | name | fw | | router_ids | cfa3e65e-d101-4cc7-80e5-39daf72c6572 | | status | CREATED | | tenant_id | fc170b1b8a9a467b9e1a63d85ced5a86 | +--------------------+--------------------------------------+ $ neutron firewall-show fw +--------------------+--------------------------------------+ | Field | Value | +--------------------+--------------------------------------+ | admin_state_up | True | | description | | | firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 | | id | afd38a9e-cf0a-4667-94e0-853a888fd981 | | name | fw | | router_ids | cfa3e65e-d101-4cc7-80e5-39daf72c6572 | | status | ACTIVE | | tenant_id | fc170b1b8a9a467b9e1a63d85ced5a86 | +--------------------+--------------------------------------+ $ neutron firewall-delete fw $ less neutron-service_error.log 2017-01-20 19:46:11.593 19338 ERROR oslo_messaging.rpc.dispatcher [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] Exception during message handling: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher fw_db = self.plugin._get_firewall(context, firewall_id) 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher raise fw_ext.FirewallNotFound(firewall_id=id) 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. 2017-01-20 19:46:11.593 19338 TRACE oslo_messaging.rpc.dispatcher 2017-01-20 19:46:11.594 19338 ERROR oslo_messaging._drivers.common [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] Returning exception Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. to caller 2017-01-20 19:46:11.594 19338 ERROR oslo_messaging._drivers.common [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] ['Traceback (most recent call last):\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatc h_and_reply\n executor_callback))\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rp c/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n fw_db = self.plugin._get_firewall(context, firewall_id)\n', ' Fi le "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found.\n'] 2017-01-20 19:46:11.595 19338 ERROR oslo_messaging.rpc.dispatcher [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] Exception during message handling: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher fw_db = self.plugin._get_firewall(context, firewall_id) 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher raise fw_ext.FirewallNotFound(firewall_id=id) 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. 2017-01-20 19:46:11.595 19338 TRACE oslo_messaging.rpc.dispatcher 2017-01-20 19:46:11.596 19338 ERROR oslo_messaging._drivers.common [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] Returning exception Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. to caller 2017-01-20 19:46:11.596 19338 ERROR oslo_messaging._drivers.common [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] ['Traceback (most recent call last):\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatc h_and_reply\n executor_callback))\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rp c/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n fw_db = self.plugin._get_firewall(context, firewall_id)\n', ' Fi le "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found.\n'] 2017-01-20 19:46:11.599 19338 ERROR oslo_messaging.rpc.dispatcher [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] Exception during message handling: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher fw_db = self.plugin._get_firewall(context, firewall_id) 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher raise fw_ext.FirewallNotFound(firewall_id=id) 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. 2017-01-20 19:46:11.599 19338 TRACE oslo_messaging.rpc.dispatcher 2017-01-20 19:46:11.600 19338 ERROR oslo_messaging._drivers.common [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] Returning exception Firewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found. to caller 2017-01-20 19:46:11.600 19338 ERROR oslo_messaging._drivers.common [req-c4af8425-b05a-4c4e-98e0-4dabe0057df7 ] ['Traceback (most recent call last):\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n fw_db = self.plugin._get_firewall(context, firewall_id)\n', ' File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Fir ewall afd38a9e-cf0a-4667-94e0-853a888fd981 could not be found.\n'] To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1658060/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
