Reviewed: https://review.opendev.org/735432
Committed:
https://git.openstack.org/cgit/openstack/neutron/commit/?id=5fdfd4cbfc56425c56b4a5702d92a97da56ab6e8
Submitter: Zuul
Branch: master
commit 5fdfd4cbfc56425c56b4a5702d92a97da56ab6e8
Author: LIU Yulong <i...@liuyulong.me>
Date: Sat Jun 13 23:14:47 2020 +0800
[L3] Check agent gateway port robustly
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError. This patch adds a simple check
for that gateway port, if it is not created, new one.
[1] https://review.opendev.org/#/c/702547/
Closes-Bug: #1883089
Change-Id: Ia90f2ee435b0a3476dbea028d3200cefe11e35e4
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1883089
Title:
[L3] floating IP failed to bind due to no agent gateway port(fip-ns)
Status in neutron:
Fix Released
Bug description:
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host
host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host
host-compute-1. Probably it was just created by other worker.
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port None found for the destination host: host-compute-1
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message
handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback
(most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in
_process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220,
in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190,
in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in
create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in
_populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in
self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in
_each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips
= port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1883089/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
