Public bug reported:
In the _post_live_migration_remove_source_vol_connections phase of live
migration, if the attachment delete failed and it will not raise
exception, but the instance's volume attachment that based source node
will remain and instance will create a new attachment record on the
destination node. So, the instance' volume will have two attachments.
And now, if we execute instance evacuation action, instance evacuate
will failed due to it's volume have two attachments.
nova raise exception as below:
2024-10-30 06:20:12.381 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.381471453+08:00 stdout F 2024-10-30 06:20:12.380 25474
ERROR nova.volume.cinder [req-43f85cdc-e536-4304-b384-663c54aaced4
6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default
default] Update attachment failed for attachment
f309ae52-3954-47b3-a43f-fed9f426853b. Error: Unable to update
attachment.(Invalid volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794) Code: 500:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid
volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794)ESC[00m
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751598438+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [req-43f85cdc-e536-4304-b384-663c54aaced4
6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default
default] [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Instance failed block
device setup: cinderclient.exceptions.ClientException: Unable to update
attachment.(Invalid volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751684297+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
Traceback (most recent call last):
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751704177+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line
3765, in _do_rebuild_instance
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751782996+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
self.driver.rebuild(**kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751796616+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/driver.py", line 337, in
rebuild
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751806716+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
raise NotImplementedError()
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751815136+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
NotImplementedError
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751822956+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751834196+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
During handling of the above exception, another exception occurred:
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751841496+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751848456+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
Traceback (most recent call last):
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751854936+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line
1992, in _prep_block_device
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751880296+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
wait_func=self._await_block_device_map_created)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751888656+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
893, in attach_block_devices
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751895956+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
_log_and_attach(device)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751903316+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
890, in _log_and_attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751909876+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
bdm.attach(*attach_args, **attach_kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751916336+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
789, in attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751923896+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
context, instance, volume_api, virt_driver)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751930455+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
50, in wrapped
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751937435+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
ret_val = method(obj, context, *args, **kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751943835+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
677, in attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.751990755+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
virt_driver, do_driver_attach)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752007575+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
664, in _do_attach
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752014735+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
do_driver_attach)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752022115+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
577, in _volume_attach
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752029415+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
self['mount_device'])['connection_info']
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752035995+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 397,
in wrapper
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752042835+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
res = method(self, ctx, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752049595+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 448,
in wrapper
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752056235+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
res = method(self, ctx, attachment_id, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752063055+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 908,
in attachment_update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752069575+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
'code': getattr(ex, 'code', None)})
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752076374+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227,
in __exit__
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752108214+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
self.force_reraise()
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752117074+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200,
in force_reraise
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752123894+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
raise self.value
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752130414+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 898,
in attachment_update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752137574+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
attachment_id, _connector)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752145134+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/api_versions.py",
line 423, in substitution
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752151634+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
return method.func(obj, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752158154+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
line 75, in update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752166374+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
resp = self._update('/attachments/%s' % id, body)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752173034+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/base.py", line 312,
in _update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752181094+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
resp, body = self.api.client.put(url, body=body, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752221713+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 220,
in put
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752272273+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
return self._cs_request(url, 'PUT', **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752283493+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 205,
in _cs_request
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752290613+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
return self.request(url, method, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752297433+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 191,
in request
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752304133+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
raise exceptions.from_response(resp, body)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752316893+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid
volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦
2024-10-30T06:20:12.752325713+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
ESC[00m
Do we have any solution for this scenario?
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2089519
Title:
instance evacuate faile due to it's volume have two attachments
Status in OpenStack Compute (nova):
New
Bug description:
In the _post_live_migration_remove_source_vol_connections phase of
live migration, if the attachment delete failed and it will not raise
exception, but the instance's volume attachment that based source node
will remain and instance will create a new attachment record on the
destination node. So, the instance' volume will have two attachments.
And now, if we execute instance evacuation action, instance evacuate
will failed due to it's volume have two attachments.
nova raise exception as below:
2024-10-30 06:20:12.381 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.381471453+08:00 stdout F 2024-10-30 06:20:12.380 25474
ERROR nova.volume.cinder [req-43f85cdc-e536-4304-b384-663c54aaced4
6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default
default] Update attachment failed for attachment
f309ae52-3954-47b3-a43f-fed9f426853b. Error: Unable to update
attachment.(Invalid volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794) Code: 500:
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid
volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794)ESC[00m
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751598438+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [req-43f85cdc-e536-4304-b384-663c54aaced4
6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default
default] [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Instance failed block
device setup: cinderclient.exceptions.ClientException: Unable to update
attachment.(Invalid volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751684297+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
Traceback (most recent call last):
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751704177+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line
3765, in _do_rebuild_instance
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751782996+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
self.driver.rebuild(**kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751796616+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/driver.py", line 337, in
rebuild
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751806716+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
raise NotImplementedError()
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751815136+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
NotImplementedError
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751822956+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751834196+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
During handling of the above exception, another exception occurred:
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751841496+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751848456+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
Traceback (most recent call last):
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751854936+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line
1992, in _prep_block_device
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751880296+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
wait_func=self._await_block_device_map_created)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751888656+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
893, in attach_block_devices
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751895956+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
_log_and_attach(device)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751903316+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
890, in _log_and_attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751909876+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
bdm.attach(*attach_args, **attach_kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751916336+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
789, in attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751923896+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
context, instance, volume_api, virt_driver)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751930455+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
50, in wrapped
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751937435+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
ret_val = method(obj, context, *args, **kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751943835+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
677, in attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.751990755+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
virt_driver, do_driver_attach)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752007575+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
664, in _do_attach
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752014735+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
do_driver_attach)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752022115+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line
577, in _volume_attach
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752029415+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
self['mount_device'])['connection_info']
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752035995+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 397,
in wrapper
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752042835+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
res = method(self, ctx, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752049595+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 448,
in wrapper
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752056235+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
res = method(self, ctx, attachment_id, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752063055+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 908,
in attachment_update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752069575+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
'code': getattr(ex, 'code', None)})
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752076374+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227,
in __exit__
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752108214+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
self.force_reraise()
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752117074+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200,
in force_reraise
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752123894+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
raise self.value
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752130414+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 898,
in attachment_update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752137574+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
attachment_id, _connector)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752145134+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/api_versions.py",
line 423, in substitution
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752151634+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
return method.func(obj, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752158154+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/v3/attachments.py",
line 75, in update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752166374+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
resp = self._update('/attachments/%s' % id, body)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752173034+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/base.py", line 312,
in _update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752181094+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
resp, body = self.api.client.put(url, body=body, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752221713+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 220,
in put
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752272273+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
return self._cs_request(url, 'PUT', **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752283493+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 205,
in _cs_request
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752290613+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
return self.request(url, method, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752297433+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 191,
in request
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752304133+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
raise exceptions.from_response(resp, body)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752316893+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid
volume: duplicate connectors detected on volume
968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID:
req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute
¦ 2024-10-30T06:20:12.752325713+08:00 stdout F 2024-10-30 06:20:12.413 25474
ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
ESC[00m
Do we have any solution for this scenario?
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2089519/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
