Reviewed: https://review.openstack.org/641480 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2a8ee40fccc65b177275d6fe80c10fdb83b86e1f Submitter: Zuul Branch: master
commit 2a8ee40fccc65b177275d6fe80c10fdb83b86e1f Author: Chris Friesen <[email protected]> Date: Wed Mar 6 14:19:01 2019 -0600 Add missing libvirt exception during device detach It turns out that when detaching a device libvirt can raise a libvirt.VIR_ERR_INTERNAL_ERROR exception with an error log of "unable to execute QEMU command 'device_del': Device <foo> not found". Add this exception to the existing "not found" case which currently handles only libvirt.VIR_ERR_OPERATION_FAILED. Change-Id: I3055cd7641de92ab188de73733ca9288a9ca730a Closes-Bug: #1815949 Signed-off-by: Chris Friesen <[email protected]> ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1815949 Title: missing special-case libvirt exception during device detach Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Bug description: In Pike a customer has run into the following issue: 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed: libvirtError: internal error: unable to execute QEMU command 'device_del': Device 'virtio-disk15' not found 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall Traceback (most recent call last): 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 143, in _run_loop 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 363, in _func 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result = f(*args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 505, in _do_wait_and_retry_detach 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall _try_detach_device(config, persistent=False, host=host) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 467, in _try_detach_device 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall device=alternative_device_name) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall self.force_reraise() 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 451, in _try_detach_device 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall self.detach_device(conf, persistent=persistent, live=live) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 530, in detach_device 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall self._domain.detachDeviceFlags(device_xml, flags=flags) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result = proxy_call(self._autowrap, f, *args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv = execute(f, *args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall six.reraise(c, e, tb) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv = meth(*args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1217, in detachDeviceFlags 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall libvirtError: internal error: unable to execute QEMU command 'device_del': Device 'virtio-disk15' not found 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall Based on discussion with Melanie Witt, it seems likely that nova is missing a special-case in Guest.detach_device_with_retry(). It seems likely we need to modify the conditional at line 409 of virt/libvirt/guest.py to look like 'if errcode in (libvirt.VIR_ERR_OPERATION_FAILED, libvirt.VIR_ERR_INTERNAL_ERROR):' To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1815949/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
