Source node
https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/subnode-2/libvirt/qemu/instance-0000001a.txt
2020-11-20 14:25:24.887+0000: starting up libvirt version: 6.0.0,
package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack-
testing-...@ubuntu.com> Tue, 15 Sep 2020 20:36:28 +0000), qemu version:
4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic,
hostname: ubuntu-bionic-ovh-bhs1-0021872195
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-19-instance-0000001a \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-19-instance-
0000001a/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-19-instance-0000001a/.cache
\
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-19-instance-
0000001a/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=instance-0000001a,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-19
-instance-0000001a/master-key.aes \
-machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \
-cpu qemu64,hypervisor=on,lahf-lm=on \
-m 128 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \
-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=22.1.0,serial=2c468d92-4b19-426a-
8c25-16b4624c21a4,uuid=2c468d92-4b19-426a-
8c25-16b4624c21a4,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79
","node-name":"libvirt-2-storage","cache":{"direct":true,"no-
flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-
only":true,"cache":{"direct":true,"no-
flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19
-426a-8c25-16b4624c21a4/disk","node-
name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false
},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-
only":false,"cache":{"direct":true,"no-
flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}'
\
-device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-
disk0,bootindex=1,write-cache=on \
-netdev tap,fd=37,id=hostnet0 \
-device virtio-net-
pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3
\
-add-fd set=2,fd=39 \
-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-vnc 127.0.0.1:0 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
-object rng-random,id=objrng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
\
-msg timestamp=on
char device redirected to /dev/pts/2 (label charserial0)
virtio: bogus descriptor or out of resources
2020-11-20 14:25:39.911+0000: initiating migration
2020-11-20T14:26:21.409517Z qemu-system-x86_64: terminating on signal 15
from pid 17395 (/usr/sbin/libvirtd)
2020-11-20 14:26:21.610+0000: shutting down, reason=destroyed
Target node
https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/libvir
t/qemu/instance-0000001a.txt
2020-11-20 14:25:11.589+0000: starting up libvirt version: 6.0.0,
package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack-
testing-...@ubuntu.com> Tue, 15 Sep 2020 20:36:28 +0000), qemu version:
4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic,
hostname: ubuntu-bionic-ovh-bhs1-0021872194
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-10-instance-0000001a \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-10-instance-
0000001a/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-10-instance-0000001a/.cache
\
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-10-instance-
0000001a/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=instance-0000001a,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10
-instance-0000001a/master-key.aes \
-machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \
-cpu qemu64 \
-m 128 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \
-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=22.1.0,serial=2c468d92-4b19-426a-
8c25-16b4624c21a4,uuid=2c468d92-4b19-426a-
8c25-16b4624c21a4,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79
","node-name":"libvirt-2-storage","cache":{"direct":true,"no-
flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-
only":true,"cache":{"direct":true,"no-
flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19
-426a-8c25-16b4624c21a4/disk","node-
name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false
},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-
only":false,"cache":{"direct":true,"no-
flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}'
\
-device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-
disk0,bootindex=1,write-cache=on \
-netdev tap,fd=34,id=hostnet0 \
-device virtio-net-
pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3
\
-add-fd set=2,fd=36 \
-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-vnc 0.0.0.0:0 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
-object rng-random,id=objrng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
\
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
2020-11-20 14:25:25.637+0000: initiating migration
2020-11-20 14:25:26.776+0000: shutting down, reason=migrated
2020-11-20T14:25:26.777394Z qemu-system-x86_64: terminating on signal 15
from pid 31113 (/usr/sbin/libvirtd)
2020-11-20 14:25:38.909+0000: starting up libvirt version: 6.0.0,
package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack-
testing-...@ubuntu.com> Tue, 15 Sep 2020 20:36:28 +0000), qemu version:
4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic,
hostname: ubuntu-bionic-ovh-bhs1-0021872194
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-13-instance-0000001a \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-13-instance-
0000001a/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-13-instance-0000001a/.cache
\
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-13-instance-
0000001a/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=instance-0000001a,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13
-instance-0000001a/master-key.aes \
-machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \
-cpu qemu64,hypervisor=on,lahf-lm=on \
-m 128 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \
-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=22.1.0,serial=2c468d92-4b19-426a-
8c25-16b4624c21a4,uuid=2c468d92-4b19-426a-
8c25-16b4624c21a4,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79
","node-name":"libvirt-2-storage","cache":{"direct":true,"no-
flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-
only":true,"cache":{"direct":true,"no-
flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-blockdev
'{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19
-426a-8c25-16b4624c21a4/disk","node-
name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false
},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-
only":false,"cache":{"direct":true,"no-
flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}'
\
-device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-
disk0,bootindex=1,write-cache=on \
-netdev tap,fd=36,id=hostnet0 \
-device virtio-net-
pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3
\
-add-fd set=2,fd=38 \
-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-vnc 0.0.0.0:0 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
-object rng-random,id=objrng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
\
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
2020-11-20T14:25:40.720757Z qemu-system-x86_64: VQ 0 size 0x80 Guest
index 0xb8 inconsistent with Host index 0xe0: delta 0xffd8
2020-11-20T14:25:40.720785Z qemu-system-x86_64: Failed to load virtio-
blk:virtio
2020-11-20T14:25:40.720790Z qemu-system-x86_64: error while loading
state for instance 0x0 of device '0000:00:04.0/virtio-blk'
2020-11-20T14:25:40.720824Z qemu-system-x86_64: load of migration
failed: Operation not permitted
2020-11-20 14:25:40.778+0000: shutting down, reason=failed
** Also affects: qemu
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1761798
Title:
live migration intermittently fails in CI with "VQ 0 size 0x80 Guest
index 0x12c inconsistent with Host index 0x134: delta 0xfff8"
Status in OpenStack Compute (nova):
Confirmed
Status in QEMU:
New
Bug description:
Seen here:
http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-
multinode-live-
migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz
2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev
pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to
/dev/pts/0 (label charserial0)
warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index
0x12c inconsistent with Host index 0x134: delta 0xfff8
2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load
virtio-blk:virtio
2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for
instance 0x0 of device '0000:00:04.0/virtio-blk'
2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed:
Operation not permitted
2018-04-05 21:48:43.198+0000: shutting down, reason=crashed
And in the n-cpu logs on the other host:
http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-
multinode-live-
migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541
There is a related Red Hat bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1450524
The CI job failures are at present using the Pike UCA:
ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0
ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions
