Public bug reported:
There seems to be a race condition when trying to add default gateway
route in fip namespace for the fip agent gateway port.
The way it happens is at high scale testing, when there is a router
update that is currently happening for the Router-A which has a
floatingip, a fip namespace is getting created and gateway ports plugged
to the external bridge in the context of the fip namespace. While it is
getting created, if there is another router update for the same
Router-A, then it calls 'update-gateway-port' and tries to set the
default gateway and fails.
We do find a log message in the l3-agent with 'Failed to process compatible
router' and also a TRACE in the l3-agent.
Traceback (most recent call last):
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 501, in _process_router_update
self._process_router_if_compatible(router)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 440, in _process_router_if_compatible
self._process_updated_router(router)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 454, in _process_updated_router
ri.process(self)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 538, in process
super(DvrLocalRouter, self).process(agent)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py",
line 31, in process
super(DvrRouterBase, self).process(agent)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py",
line 396, in call
self.logger(e)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
self.force_reraise()
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py",
line 393, in call
return func(*args, **kwargs)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 989, in process
self.process_external(agent)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 491, in process_external
self.create_dvr_fip_interfaces(ex_gw_port)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 522, in create_dvr_fip_interfaces
self.fip_ns.update_gateway_port(fip_agent_port)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py",
line 243, in update_gateway_port
ipd.route.add_gateway(gw_ip)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 690, in add_gateway
self._as_root([ip_version], tuple(args))
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 361, in _as_root
use_root_namespace=use_root_namespace)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 94, in _as_root
log_fail_as_error=self.log_fail_as_error)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 103, in _execute
log_fail_as_error=log_fail_as_error)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
line 140, in execute
raise RuntimeError(msg)
** Affects: neutron
Importance: Undecided
Status: New
** Tags: l3-dvr-backlog mitaka-backport-potential newton-backport-potential
** Summary changed:
- Fix race conditions when trying to add default gateway for fip gateway port.
+ DVR: Fix race conditions when trying to add default gateway for fip gateway
port.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1631513
Title:
DVR: Fix race conditions when trying to add default gateway for fip
gateway port.
Status in neutron:
New
Bug description:
There seems to be a race condition when trying to add default gateway
route in fip namespace for the fip agent gateway port.
The way it happens is at high scale testing, when there is a router
update that is currently happening for the Router-A which has a
floatingip, a fip namespace is getting created and gateway ports
plugged to the external bridge in the context of the fip namespace.
While it is getting created, if there is another router update for the
same Router-A, then it calls 'update-gateway-port' and tries to set
the default gateway and fails.
We do find a log message in the l3-agent with 'Failed to process compatible
router' and also a TRACE in the l3-agent.
Traceback (most recent call last):
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 501, in _process_router_update
self._process_router_if_compatible(router)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 440, in _process_router_if_compatible
self._process_updated_router(router)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 454, in _process_updated_router
ri.process(self)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 538, in process
super(DvrLocalRouter, self).process(agent)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py",
line 31, in process
super(DvrRouterBase, self).process(agent)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py",
line 396, in call
self.logger(e)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
self.force_reraise()
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py",
line 393, in call
return func(*args, **kwargs)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 989, in process
self.process_external(agent)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 491, in process_external
self.create_dvr_fip_interfaces(ex_gw_port)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 522, in create_dvr_fip_interfaces
self.fip_ns.update_gateway_port(fip_agent_port)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py",
line 243, in update_gateway_port
ipd.route.add_gateway(gw_ip)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 690, in add_gateway
self._as_root([ip_version], tuple(args))
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 361, in _as_root
use_root_namespace=use_root_namespace)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 94, in _as_root
log_fail_as_error=self.log_fail_as_error)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py",
line 103, in _execute
log_fail_as_error=log_fail_as_error)
File
"/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
line 140, in execute
raise RuntimeError(msg)
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1631513/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
