** Also affects: nova/icehouse
Importance: Undecided
Status: New
** Changed in: nova/icehouse
Status: New => Fix Committed
** Changed in: nova/icehouse
Milestone: None => 2014.1.1
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1288392
Title:
instances get stuck in ERROR/deleting when Neutron is unavailable
Status in OpenStack Compute (Nova):
Fix Committed
Status in OpenStack Compute (nova) icehouse series:
Fix Committed
Bug description:
We had a temporary outage of Neutron, and many instances got stuck in
this state. 'nova delete' on them does not work until nova-compute is
forcibly restarted.
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value
|
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host |
ci-overcloud-novacompute1-4q2dbhdklrkq
|
| OS-EXT-SRV-ATTR:hypervisor_hostname |
ci-overcloud-novacompute1-4q2dbhdklrkq.novalocal
|
| OS-EXT-SRV-ATTR:instance_name | instance-00003f80
|
| OS-EXT-STS:power_state | 1
|
| OS-EXT-STS:task_state | deleting
|
| OS-EXT-STS:vm_state | error
|
| OS-SRV-USG:launched_at | 2014-03-05T03:54:49.000000
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| config_drive |
|
| created | 2014-03-05T03:46:25Z
|
| default-net network | 10.0.58.225
|
| fault | {"message": "Connection to neutron
failed: Maximum attempts reached", "code": 500, "details": " File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 253, in decorated_function |
| | return function(self, context,
*args, **kwargs)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 2038, in terminate_instance
|
| | do_terminate_instance(instance,
bdms)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py\",
line 249, in inner
|
| | return f(*args, **kwargs)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 2036, in do_terminate_instance
|
| |
self._set_instance_error_state(context, instance['uuid'])
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\",
line 68, in __exit__
|
| | six.reraise(self.type_,
self.value, self.tb)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 2026, in do_terminate_instance
|
| | reservations=reservations)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/hooks.py\", line
103, in inner
|
| | rv = f(*args, **kwargs)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 2005, in _delete_instance
|
| | user_id=user_id)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\",
line 68, in __exit__
|
| | six.reraise(self.type_,
self.value, self.tb)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 1975, in _delete_instance
|
| | self._shutdown_instance(context,
db_inst, bdms)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 1884, in _shutdown_instance
|
| | network_info =
self._get_instance_nw_info(context, instance)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\",
line 902, in _get_instance_nw_info
|
| | instance)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/api.py\",
line 48, in wrapper
|
| | res = f(self, context, *args,
**kwargs)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\",
line 445, in get_instance_nw_info
|
| | result =
self._get_instance_nw_info(context, instance, networks)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\",
line 452, in _get_instance_nw_info
|
| | nw_info =
self._build_network_info_model(context, instance, networks)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\",
line 1010, in _build_network_info_model
|
| | data =
client.list_ports(**search_opts)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\",
line 112, in with_params
|
| | ret = self.function(instance,
*args, **kwargs)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\",
line 307, in list_ports
|
| | **_params)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\",
line 1251, in list
|
| | for r in
self._pagination(collection, path, **params):
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\",
line 1264, in _pagination
|
| | res = self.get(path,
params=params)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\",
line 1237, in get
|
| | headers=headers, params=params)
|
| | File
\"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\",
line 1229, in retry_request
|
| | raise
exceptions.ConnectionFailed(reason=_(\"Maximum attempts reached\"))
|
| | ", "created":
"2014-03-05T04:22:04Z"}
|
| flavor | h1.large
(872d8f61-c45a-45c3-87da-466d9f0f241b)
|
| hostId |
0bab209cc6f26a8d5c4bc76e3da39d4fa68e5fefc6e5c0eada7a90d2
|
| id | ae9c75e3-51d2-43a3-8b20-34375b4c72d3
|
| image |
tripleo-precise-1393812840.template.openstack.org
(114e4b92-567e-4348-9ed8-e88281104208)
|
| key_name | -
|
| metadata | {}
|
| name |
tripleo-precise-tripleo-test-cloud-2125650.slave.openstack.org
|
| os-extended-volumes:volumes_attached | []
|
| security_groups | default, default
|
| status | ERROR
|
| tenant_id | 64d2d3bc07084ef1accd4e3502909c77
|
| tripleo-bm-test network | 192.168.1.78
|
| updated | 2014-03-05T04:22:04Z
|
| user_id | 35ef3ce265cb4a25b5303f3daa143f4e
|
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1288392/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
