Public bug reported:
If user was capable to create broken network configuration, instance
become undeletable. Reason why user can create broken networking is
under investigation (current hypothesis: if network (neutron) created in
one tennant and instance in other, and user is admin in both tenants, it
cause broken configuration).
But such instance deletetion cause trace on nova-compute:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
line 461, in _process_data
**args)
File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 172, in dispatch
result = getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in
decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
payload)
File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in
decorated_function
pass
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in
decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in
decorated_function
function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in
decorated_function
e, sys.exc_info())
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in
decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792,
in terminate_instance
do_terminate_instance(instance, bdms)
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
line 246, in inner
return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784,
in do_terminate_instance
reservations=reservations)
File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
rv = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757,
in _delete_instance
user_id=user_id)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729,
in _delete_instance
self._shutdown_instance(context, db_inst, bdms)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639,
in _shutdown_instance
network_info = self._get_instance_nw_info(context, instance)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876, in
_get_instance_nw_info
instance)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
455, in get_instance_nw_info
result = self._get_instance_nw_info(context, instance, networks)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
463, in _get_instance_nw_info
nw_info = self._build_network_info_model(context, instance, networks)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
1009, in _build_network_info_model
subnets)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
962, in _nw_info_build_network
label=network_name,
UnboundLocalError: local variable 'network_name' referenced before assignment
The reason is following code :
def _nw_info_build_network(self, port, networks, subnets):
# NOTE(danms): This loop can't fail to find a network since we
# filtered ports to only the ones matching networks in our parent
for net in networks:
if port['network_id'] == net['id']:
network_name = net['name']
break
(if no net found network_name become undefined).
Following patch should allow instance deletion in case of networking
problems:
diff --git a/nova/network/neutronv2/api.py b/nova/network/neutronv2/api.py
index a41924d..8a44f99 100644
--- a/nova/network/neutronv2/api.py
+++ b/nova/network/neutronv2/api.py
@@ -939,6 +939,8 @@ class API(base.Base):
if port['network_id'] == net['id']:
network_name = net['name']
break
+ else:
+ network_name = ""
bridge = None
ovs_interfaceid = None
** Affects: nova
Importance: Undecided
Status: New
** Attachment added: "Fix deletion of instances with broken networking"
https://bugs.launchpad.net/bugs/1271958/+attachment/3955123/+files/nova-compute-fix-broken-net-instance-deletion.patch
** Description changed:
If user was capable to create broken network configuration, instance
become undeletable. Reason why user can create broken networking is
under investigation (current hypothesis: if network (neutron) created in
one tennant and instance in other, and user is admin in both tenants, it
cause broken configuration).
But such instance deletetion cause trace on nova-compute:
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp Traceback
(most recent call last):
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461,
in _process_data
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp **args)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 172, in dispatch
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp result
= getattr(proxyobj, method)(ctxt, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in
decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp payload)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp return
f(self, context, *args, **kw)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in
decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp pass
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in
decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in
decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in
decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp e,
sys.exc_info())
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in
decorated_function
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792, in
terminate_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
do_terminate_instance(instance, bdms)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
246, in inner
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp return
f(*args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784, in
do_terminate_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
reservations=reservations)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp rv =
f(*args, **kwargs)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757, in
_delete_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
user_id=user_id)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729, in
_delete_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
self._shutdown_instance(context, db_inst, bdms)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639, in
_shutdown_instance
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
network_info = self._get_instance_nw_info(context, instance)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876, in
_get_instance_nw_info
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
instance)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 455, in
get_instance_nw_info
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp result
= self._get_instance_nw_info(context, instance, networks)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 463, in
_get_instance_nw_info
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp nw_info
= self._build_network_info_model(context, instance, networks)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1009, in
_build_network_info_model
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp subnets)
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 962, in
_nw_info_build_network
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
label=network_name,
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
UnboundLocalError: local variable 'network_name' referenced before assignment
- 2014-01-23 15:43:18.249 1411 TRACE nova.openstack.common.rpc.amqp
+ Traceback (most recent call last):
+ File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
line 461, in _process_data
+ **args)
+ File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 172, in dispatch
+ result = getattr(proxyobj, method)(ctxt, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353,
in decorated_function
+ return function(self, context, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in
wrapped
+ payload)
+ File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in
wrapped
+ return f(self, context, *args, **kw)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243,
in decorated_function
+ pass
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229,
in decorated_function
+ return function(self, context, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294,
in decorated_function
+ function(self, context, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271,
in decorated_function
+ e, sys.exc_info())
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258,
in decorated_function
+ return function(self, context, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792,
in terminate_instance
+ do_terminate_instance(instance, bdms)
+ File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
line 246, in inner
+ return f(*args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784,
in do_terminate_instance
+ reservations=reservations)
+ File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
+ rv = f(*args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757,
in _delete_instance
+ user_id=user_id)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729,
in _delete_instance
+ self._shutdown_instance(context, db_inst, bdms)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639,
in _shutdown_instance
+ network_info = self._get_instance_nw_info(context, instance)
+ File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876,
in _get_instance_nw_info
+ instance)
+ File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
455, in get_instance_nw_info
+ result = self._get_instance_nw_info(context, instance, networks)
+ File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
463, in _get_instance_nw_info
+ nw_info = self._build_network_info_model(context, instance, networks)
+ File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
1009, in _build_network_info_model
+ subnets)
+ File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
962, in _nw_info_build_network
+ label=network_name,
+ UnboundLocalError: local variable 'network_name' referenced before assignment
- The reason is following code:
+ The reason is following code :
- def _nw_info_build_network(self, port, networks, subnets):
- # NOTE(danms): This loop can't fail to find a network since we
- # filtered ports to only the ones matching networks in our parent
- for net in networks:
- if port['network_id'] == net['id']:
- network_name = net['name']
- break
+ def _nw_info_build_network(self, port, networks, subnets):
+ # NOTE(danms): This loop can't fail to find a network since we
+ # filtered ports to only the ones matching networks in our parent
+ for net in networks:
+ if port['network_id'] == net['id']:
+ network_name = net['name']
+ break
(if no net found network_name become undefined).
Following patch should allow instance deletion in case of networking
problems:
diff --git a/nova/network/neutronv2/api.py b/nova/network/neutronv2/api.py
index a41924d..8a44f99 100644
--- a/nova/network/neutronv2/api.py
+++ b/nova/network/neutronv2/api.py
@@ -939,6 +939,8 @@ class API(base.Base):
- if port['network_id'] == net['id']:
- network_name = net['name']
- break
+ if port['network_id'] == net['id']:
+ network_name = net['name']
+ break
+ else:
+ network_name = ""
-
- bridge = None
- ovs_interfaceid = None
+
+ bridge = None
+ ovs_interfaceid = None
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1271958
Title:
nova compute fail to remove instance with port if network is broken
Status in OpenStack Compute (Nova):
New
Bug description:
If user was capable to create broken network configuration, instance
become undeletable. Reason why user can create broken networking is
under investigation (current hypothesis: if network (neutron) created
in one tennant and instance in other, and user is admin in both
tenants, it cause broken configuration).
But such instance deletetion cause trace on nova-compute:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
line 461, in _process_data
**args)
File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 172, in dispatch
result = getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in
wrapped
payload)
File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in
wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243,
in decorated_function
pass
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294,
in decorated_function
function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271,
in decorated_function
e, sys.exc_info())
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1792,
in terminate_instance
do_terminate_instance(instance, bdms)
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
line 246, in inner
return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1784,
in do_terminate_instance
reservations=reservations)
File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
rv = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1757,
in _delete_instance
user_id=user_id)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1729,
in _delete_instance
self._shutdown_instance(context, db_inst, bdms)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1639,
in _shutdown_instance
network_info = self._get_instance_nw_info(context, instance)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 876,
in _get_instance_nw_info
instance)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
455, in get_instance_nw_info
result = self._get_instance_nw_info(context, instance, networks)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
463, in _get_instance_nw_info
nw_info = self._build_network_info_model(context, instance, networks)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
1009, in _build_network_info_model
subnets)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
962, in _nw_info_build_network
label=network_name,
UnboundLocalError: local variable 'network_name' referenced before assignment
The reason is following code :
def _nw_info_build_network(self, port, networks, subnets):
# NOTE(danms): This loop can't fail to find a network since we
# filtered ports to only the ones matching networks in our parent
for net in networks:
if port['network_id'] == net['id']:
network_name = net['name']
break
(if no net found network_name become undefined).
Following patch should allow instance deletion in case of networking
problems:
diff --git a/nova/network/neutronv2/api.py b/nova/network/neutronv2/api.py
index a41924d..8a44f99 100644
--- a/nova/network/neutronv2/api.py
+++ b/nova/network/neutronv2/api.py
@@ -939,6 +939,8 @@ class API(base.Base):
if port['network_id'] == net['id']:
network_name = net['name']
break
+ else:
+ network_name = ""
bridge = None
ovs_interfaceid = None
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1271958/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
