** Also affects: cinder
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2077512
Title:
resize doesnt validate state on volumes, instance set to error when
attachment fails
Status in Cinder:
New
Status in OpenStack Compute (nova):
In Progress
Bug description:
nova 26.2.1 (zed) looks like same code path in master
If we issue a resize the _terminate_volume_connections() function in
nova/compute/manager.py will call cinder to delete the attachments,
but we never check the actual volume status before doing this so if
the volume(s) is in a invalid state we will ERROR (because we are in a
_error_out_instance_on_exception context) the instance and only an
cloud admin can restore it.
We should do either of:
* Validate volume status before doing a resize so that we don't fail (which
could cause a race condition between check and actual attachment)
* We could handle the error and raise a InstanceFaultRollback and since we
are in a _error_out_instance_on_exception context to reset the instance back to
it's status, then instead set the error details on the instance or add a
instance log entry
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [None
req-7b16c6f1-ccd8-46c8-a4c6-9b6a5b716233
833c476e9c1adb00f53b90cfaa50d077788946d8cd916606dee388eaaba2a71f
3b680fb64bc848be9b647895c8f89cdb - - 6ee763824d0145eeb1174bf30416df2d default]
[instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] Setting instance vm_state to
ERROR: nova.exception.InvalidInput: Invalid input received: Invalid volume:
Volume 33025874-2586-4687-ae9c-478d291a0381 status must be available or
downloading to reserve, but the current status is backing-up. (HTTP 400)
(Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] Traceback (most recent call last):
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 397, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] res = method(self, ctx, *args,
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 435, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] res = method(self, ctx, volume_id,
*args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 810, in
attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] LOG.error('Create attachment failed
for volume '
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] self.force_reraise()
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] raise self.value
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 802, in
attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] attachment_ref =
cinderclient(context, '3.44').attachments.create(
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in
substitution
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] return method.func(obj, *args,
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 39, in
create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] retval = self._create('/attachments',
body, 'attachment')
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/base.py", line 306, in _create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] resp, body =
self.api.client.post(url, body=body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/client.py", line 227, in post
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] return self._cs_request(url, 'POST',
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/client.py", line 215, in
_cs_request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] return self.request(url, method,
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/client.py", line 201, in request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] raise exceptions.from_response(resp,
body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] cinderclient.exceptions.BadRequest:
Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381 status must be
available or downloading to reserve, but the current status is backing-up.
(HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8]
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] During handling of the above exception,
another exception occurred:
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8]
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] Traceback (most recent call last):
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10612, in
_error_out_instance_on_exception
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] yield
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5896, in
_resize_instance
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8]
self._terminate_volume_connections(context, instance, bdms)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5964, in
_terminate_volume_connections
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] attachment_id =
self.volume_api.attachment_create(
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 405, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8]
_reraise(exception.InvalidInput(reason=err_msg))
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 488, in _reraise
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] raise
desired_exc.with_traceback(sys.exc_info()[2])
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 397, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] res = method(self, ctx, *args,
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 435, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] res = method(self, ctx, volume_id,
*args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 810, in
attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] LOG.error('Create attachment failed
for volume '
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] self.force_reraise()
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] raise self.value
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 802, in
attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] attachment_ref =
cinderclient(context, '3.44').attachments.create(
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in
substitution
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] return method.func(obj, *args,
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 39, in
create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] retval = self._create('/attachments',
body, 'attachment')
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/base.py", line 306, in _create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] resp, body =
self.api.client.post(url, body=body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/client.py", line 227, in post
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] return self._cs_request(url, 'POST',
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/client.py", line 215, in
_cs_request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] return self.request(url, method,
**kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] File
"/usr/lib/python3.9/site-packages/cinderclient/client.py", line 201, in request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] raise exceptions.from_response(resp,
body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance:
a7fae912-b312-4d3e-9963-4d3b06f705f8] nova.exception.InvalidInput: Invalid
input received: Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381
status must be available or downloading to reserve, but the current status is
backing-up. (HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/2077512/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
