Public bug reported: Seeing this failure in the gate:
https://zuul.opendev.org/t/openstack/build/7c71502b04fe47039b87f76fbe04fe56/log/controller/logs/screen-n-cpu.txt#33096 Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [req-77f51485-cbc2-4f2a-8a6d-4a8ed910e585 req-a221d4f9-401e-420a-911e-8d32536a1d23 service nova] [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] detaching network adapter failed.: libvirt.libvirtError: internal error: End of file from qemu monitor Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] Traceback (most recent call last): Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2210, in detach_interface Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] wait_for_detach = guest.detach_device_with_retry( Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 423, in detach_device_with_retry Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] _try_detach_device(conf, persistent, live) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 412, in _try_detach_device Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] ctx.reraise = True Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/oslo_utils/excutils.py", line 220, in __exit__ Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] self.force_reraise() Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/oslo_utils/excutils.py", line 196, in force_reraise Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] six.reraise(self.type_, self.value, self.tb) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/six.py", line 703, in reraise Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] raise value Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 398, in _try_detach_device Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] self.detach_device(conf, persistent=persistent, live=live) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 473, in detach_device Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] self._domain.detachDeviceFlags(device_xml, flags=flags) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/eventlet/tpool.py", line 190, in doit Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] result = proxy_call(self._autowrap, f, *args, **kwargs) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/eventlet/tpool.py", line 148, in proxy_call Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] rv = execute(f, *args, **kwargs) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/eventlet/tpool.py", line 129, in execute Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] six.reraise(c, e, tb) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/six.py", line 703, in reraise Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] raise value Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/eventlet/tpool.py", line 83, in tworker Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] rv = meth(*args, **kwargs) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8/dist- packages/libvirt.py", line 1515, in detachDeviceFlags Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] raise libvirtError('virDomainDetachDeviceFlags() failed') Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] libvirt.libvirtError: internal error: End of file from qemu monitor Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova- compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c- 8cf4-46c7-9498-2c5dbc452431] This was in a cinder-glance-multistore test, but I'm not sure it has anything to do with those. That test backs glance with cinder, but otherwise I wouldn't think nova would notice unless it was a failure to snapshot, use a volume, etc. ** Affects: nova Importance: Undecided Status: New ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1914664 Title: QEMU monitor read failure in ServerStableDeviceRescueTest Status in OpenStack Compute (nova): New Bug description: Seeing this failure in the gate: https://zuul.opendev.org/t/openstack/build/7c71502b04fe47039b87f76fbe04fe56/log/controller/logs/screen-n-cpu.txt#33096 Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [req-77f51485-cbc2-4f2a-8a6d-4a8ed910e585 req-a221d4f9-401e-420a-911e-8d32536a1d23 service nova] [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] detaching network adapter failed.: libvirt.libvirtError: internal error: End of file from qemu monitor Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] Traceback (most recent call last): Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2210, in detach_interface Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] wait_for_detach = guest.detach_device_with_retry( Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 423, in detach_device_with_retry Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] _try_detach_device(conf, persistent, live) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 412, in _try_detach_device Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] ctx.reraise = True Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] self.force_reraise() Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] six.reraise(self.type_, self.value, self.tb) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/six.py", line 703, in reraise Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] raise value Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 398, in _try_detach_device Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] self.detach_device(conf, persistent=persistent, live=live) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 473, in detach_device Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] self._domain.detachDeviceFlags(device_xml, flags=flags) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/eventlet/tpool.py", line 190, in doit Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] result = proxy_call(self._autowrap, f, *args, **kwargs) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/eventlet/tpool.py", line 148, in proxy_call Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] rv = execute(f, *args, **kwargs) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/eventlet/tpool.py", line 129, in execute Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] six.reraise(c, e, tb) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/six.py", line 703, in reraise Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] raise value Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/eventlet/tpool.py", line 83, in tworker Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] rv = meth(*args, **kwargs) Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] File "/usr/local/lib/python3.8 /dist-packages/libvirt.py", line 1515, in detachDeviceFlags Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] raise libvirtError('virDomainDetachDeviceFlags() failed') Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] libvirt.libvirtError: internal error: End of file from qemu monitor Feb 04 20:54:32.857198 ubuntu-focal-limestone-regionone-0022873642 nova-compute[90163]: ERROR nova.virt.libvirt.driver [instance: 7174e97c-8cf4-46c7-9498-2c5dbc452431] This was in a cinder-glance-multistore test, but I'm not sure it has anything to do with those. That test backs glance with cinder, but otherwise I wouldn't think nova would notice unless it was a failure to snapshot, use a volume, etc. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1914664/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
