Public bug reported:
When we delete a firewall under l3-ha mode, some neutron-servers reports
error logs: 'FirewallNotFound':
Environment:
* Openstack Kilo version
* Three neutron servers using ha-proxy in balance roundrobin mode, and provides
VIP (keepalived)
* l3_ha=True is set in neutron-servers to provide L3 HA.
* Three l3-agents on 3 network nodes
We found 2 out of 3 neutron-servers print the following error logs
-----------
Error logs:
===========
2016-09-12 14:33:34.250 22722 DEBUG oslo_messaging._drivers.amqp [-] unpacked
context: {u'read_deleted': u'no', u'project_name': u'zhaoyi', u'user_id':
u'5f228382dd8d4001bd079cfab624e870', u'roles': [u'_member_', u'Member',
u'user', u'admin'], u'tenant_id': u'd3147020bd1f4a709654b7e62885bd9f',
u'auth_token': u'***', u'request_id':
u'req-89116778-b4fb-4232-8249-500f1db5d3f8', u'is_admin': True, u'user':
u'5f228382dd8d4001bd079cfab624e870', u'timestamp': u'2016-09-12
06:33:33.570294', u'tenant_name': u'zhaoyi', u'project_id':
u'd3147020bd1f4a709654b7e62885bd9f', u'user_name': u'zhaoyi', u'tenant':
u'd3147020bd1f4a709654b7e62885bd9f'} unpack_context
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:203
2016-09-12 14:33:34.253 22722 DEBUG
neutron_fwaas.services.firewall.fwaas_plugin
[req-89116778-b4fb-4232-8249-500f1db5d3f8 ] firewall_deleted() called
firewall_deleted
/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py:62
2016-09-12 14:33:34.260 22722 DEBUG neutron_fwaas.db.firewall.firewall_db
[req-89116778-b4fb-4232-8249-500f1db5d3f8 ] delete_firewall() called
delete_firewall
/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:362
2016-09-12 14:33:34.303 22759 DEBUG keystoneclient.session [-] RESP: [200]
content-length: 3084 x-subject-token:
{SHA1}fb0b83f9b5d3a4b459a1f2845c1a1bd4bba3c008 vary: X-Auth-Token date: Mon, 12
Sep 2016 06:33:34 GMT content-type: application/json x-openstack-request-id:
req-38cd15ab-f193-4d04-80bd-088638497b26
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id":
"9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id":
"bf49f6a34f5b4ec8843efee8a840a8b3", "name": "Member"}, {"id":
"a104bc435a5d4031b9712dd702cb8672", "name": "user"}, {"id":
"11b9aa45b311407ba9460b95eb1534c2", "name": "admin"}], "expires_at":
"2016-09-12T07:03:12.000000Z", "project": {"domain": {"id": "default", "name":
"Default"}, "id": "d3147020bd1f4a709654b7e62885bd9f", "name": "zhaoyi"},
"catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default",
"name": "Default"}, "id": "5f228382dd8d4001bd079cfab624e870", "name":
"zhaoyi"}, "audit_ids": ["Q9kbnQe8SzOcZ7ig6_-2ew", "guy9k1VsThmCGwfavbmvjw"],
"issued_at": "2016-09-12T06:03:12.784462"}}
_http_log_response
/usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2016-09-12 14:33:34.307 22759 DEBUG neutron_fwaas.db.firewall.firewall_db
[req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] get_firewall_rules() called
get_firewall_rules
/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:534
2016-09-12 14:33:34.322 22759 INFO neutron.wsgi
[req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] 10.65.0.99,10.65.0.99 - -
[12/Sep/2016 14:33:34] "GET /v2.0/fw/firewall_rules.json HTTP/1.1" 200 6555
0.163946
2016-09-12 14:33:34.447 22722 ERROR oslo_messaging.rpc.dispatcher
[req-89116778-b4fb-4232-8249-500f1db5d3f8 ] Exception during message handling:
Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be found.
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher Traceback
(most recent call last):
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142,
in _dispatch_and_reply
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
executor_callback))
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186,
in _dispatch
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
executor_callback)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130,
in _do_dispatch
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher result =
func(ctxt, **new_args)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py",
line 67, in firewall_deleted
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
self.plugin.delete_db_firewall_object(context, firewall_id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py",
line 323, in delete_db_firewall_object
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
super(FirewallPlugin, self).delete_firewall(context, id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py",
line 368, in delete_firewall
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher raise
fw_ext.FirewallNotFound(firewall_id=id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
FirewallNotFound: Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be
found.
** Affects: neutron
Importance: Undecided
Status: New
** Tags: fwaas sg-fw
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1622460
Title:
neutron-server report 'FirewallNotFound' when delete firewall under
l3_ha mode
Status in neutron:
New
Bug description:
When we delete a firewall under l3-ha mode, some neutron-servers
reports error logs: 'FirewallNotFound':
Environment:
* Openstack Kilo version
* Three neutron servers using ha-proxy in balance roundrobin mode, and
provides VIP (keepalived)
* l3_ha=True is set in neutron-servers to provide L3 HA.
* Three l3-agents on 3 network nodes
We found 2 out of 3 neutron-servers print the following error logs
-----------
Error logs:
===========
2016-09-12 14:33:34.250 22722 DEBUG oslo_messaging._drivers.amqp [-] unpacked
context: {u'read_deleted': u'no', u'project_name': u'zhaoyi', u'user_id':
u'5f228382dd8d4001bd079cfab624e870', u'roles': [u'_member_', u'Member',
u'user', u'admin'], u'tenant_id': u'd3147020bd1f4a709654b7e62885bd9f',
u'auth_token': u'***', u'request_id':
u'req-89116778-b4fb-4232-8249-500f1db5d3f8', u'is_admin': True, u'user':
u'5f228382dd8d4001bd079cfab624e870', u'timestamp': u'2016-09-12
06:33:33.570294', u'tenant_name': u'zhaoyi', u'project_id':
u'd3147020bd1f4a709654b7e62885bd9f', u'user_name': u'zhaoyi', u'tenant':
u'd3147020bd1f4a709654b7e62885bd9f'} unpack_context
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:203
2016-09-12 14:33:34.253 22722 DEBUG
neutron_fwaas.services.firewall.fwaas_plugin
[req-89116778-b4fb-4232-8249-500f1db5d3f8 ] firewall_deleted() called
firewall_deleted
/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py:62
2016-09-12 14:33:34.260 22722 DEBUG neutron_fwaas.db.firewall.firewall_db
[req-89116778-b4fb-4232-8249-500f1db5d3f8 ] delete_firewall() called
delete_firewall
/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:362
2016-09-12 14:33:34.303 22759 DEBUG keystoneclient.session [-] RESP: [200]
content-length: 3084 x-subject-token:
{SHA1}fb0b83f9b5d3a4b459a1f2845c1a1bd4bba3c008 vary: X-Auth-Token date: Mon, 12
Sep 2016 06:33:34 GMT content-type: application/json x-openstack-request-id:
req-38cd15ab-f193-4d04-80bd-088638497b26
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id":
"9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id":
"bf49f6a34f5b4ec8843efee8a840a8b3", "name": "Member"}, {"id":
"a104bc435a5d4031b9712dd702cb8672", "name": "user"}, {"id":
"11b9aa45b311407ba9460b95eb1534c2", "name": "admin"}], "expires_at":
"2016-09-12T07:03:12.000000Z", "project": {"domain": {"id": "default", "name":
"Default"}, "id": "d3147020bd1f4a709654b7e62885bd9f", "name": "zhaoyi"},
"catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default",
"name": "Default"}, "id": "5f228382dd8d4001bd079cfab624e870", "name":
"zhaoyi"}, "audit_ids": ["Q9kbnQe8SzOcZ7ig6_-2ew", "guy9k1VsThmCGwfavbmvjw"],
"issued_at": "2016-09-12T06:03:12.784462"}}
_http_log_response
/usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2016-09-12 14:33:34.307 22759 DEBUG neutron_fwaas.db.firewall.firewall_db
[req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] get_firewall_rules() called
get_firewall_rules
/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:534
2016-09-12 14:33:34.322 22759 INFO neutron.wsgi
[req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] 10.65.0.99,10.65.0.99 - -
[12/Sep/2016 14:33:34] "GET /v2.0/fw/firewall_rules.json HTTP/1.1" 200 6555
0.163946
2016-09-12 14:33:34.447 22722 ERROR oslo_messaging.rpc.dispatcher
[req-89116778-b4fb-4232-8249-500f1db5d3f8 ] Exception during message handling:
Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be found.
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher Traceback
(most recent call last):
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142,
in _dispatch_and_reply
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
executor_callback))
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186,
in _dispatch
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
executor_callback)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130,
in _do_dispatch
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher result
= func(ctxt, **new_args)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py",
line 67, in firewall_deleted
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
self.plugin.delete_db_firewall_object(context, firewall_id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py",
line 323, in delete_db_firewall_object
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
super(FirewallPlugin, self).delete_firewall(context, id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py",
line 368, in delete_firewall
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher raise
fw_ext.FirewallNotFound(firewall_id=id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher
FirewallNotFound: Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be
found.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1622460/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
