Public bug reported:
In Pike a customer has run into the following issue:
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall [-] Dynamic
interval looping call 'oslo_service.loopingcall._func' failed: libvirtError:
internal error: unable to execute QEMU command 'device_del': Device
'virtio-disk15' not found
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall Traceback (most
recent call last):
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 143, in
_run_loop
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result =
func(*self.args, **self.kw)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 363, in
_func
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result =
f(*args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 505, in
_do_wait_and_retry_detach
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
_try_detach_device(config, persistent=False, host=host)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 467, in
_try_detach_device
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
device=alternative_device_name)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
self.force_reraise()
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
six.reraise(self.type_, self.value, self.tb)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 451, in
_try_detach_device
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
self.detach_device(conf, persistent=persistent, live=live)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 530, in
detach_device
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
self._domain.detachDeviceFlags(device_xml, flags=flags)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result =
proxy_call(self._autowrap, f, *args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv =
execute(f, *args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall six.reraise(c,
e, tb)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv =
meth(*args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1217, in detachDeviceFlags
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall if ret == -1:
raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall libvirtError:
internal error: unable to execute QEMU command 'device_del': Device
'virtio-disk15' not found
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
Based on discussion with Melanie Witt, it seems likely that nova is missing a
special-case in Guest.detach_device_with_retry(). It seems likely we need to
modify the conditional at line 409 of virt/libvirt/guest.py to look like 'if
errcode in (libvirt.VIR_ERR_OPERATION_FAILED, libvirt.VIR_ERR_INTERNAL_ERROR):'
** Affects: nova
Importance: Undecided
Status: New
** Tags: compute
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1815949
Title:
missing special-case libvirt exception during device detach
Status in OpenStack Compute (nova):
New
Bug description:
In Pike a customer has run into the following issue:
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall [-] Dynamic
interval looping call 'oslo_service.loopingcall._func' failed: libvirtError:
internal error: unable to execute QEMU command 'device_del': Device
'virtio-disk15' not found
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall Traceback (most
recent call last):
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 143, in
_run_loop
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result =
func(*self.args, **self.kw)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 363, in
_func
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result =
f(*args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 505, in
_do_wait_and_retry_detach
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
_try_detach_device(config, persistent=False, host=host)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 467, in
_try_detach_device
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
device=alternative_device_name)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
self.force_reraise()
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
six.reraise(self.type_, self.value, self.tb)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 451, in
_try_detach_device
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
self.detach_device(conf, persistent=persistent, live=live)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 530, in
detach_device
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
self._domain.detachDeviceFlags(device_xml, flags=flags)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result =
proxy_call(self._autowrap, f, *args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv =
execute(f, *args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
six.reraise(c, e, tb)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv =
meth(*args, **kwargs)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1217, in detachDeviceFlags
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall if ret ==
-1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall libvirtError:
internal error: unable to execute QEMU command 'device_del': Device
'virtio-disk15' not found
2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall
Based on discussion with Melanie Witt, it seems likely that nova is missing a
special-case in Guest.detach_device_with_retry(). It seems likely we need to
modify the conditional at line 409 of virt/libvirt/guest.py to look like 'if
errcode in (libvirt.VIR_ERR_OPERATION_FAILED, libvirt.VIR_ERR_INTERNAL_ERROR):'
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1815949/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
