Package: nova-compute
Version: 2:18.1.0-6+deb10u2
Dear maintainer,
When I try to create and boot a new instance, the following problem arises:
2023-03-14 09:47:40.446 8685 INFO nova.compute.claims
[req-aa77422c-3d55-43b9-b819-690c6e59117b
a1434de441ca4077b26c9696b06a4cea 12e4f437b56a4badb839b816d6848d0a -
default default] [instance: eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4]
Claim successful on node ehv-ic0-hv5.openstack
2023-03-14 09:47:42.712 8685 INFO nova.virt.libvirt.driver
[req-aa77422c-3d55-43b9-b819-690c6e59117b
a1434de441ca4077b26c9696b06a4cea 12e4f437b56a4badb839b816d6848d0a -
default default] [instance: eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4]
Creating image
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager
[req-aa77422c-3d55-43b9-b819-690c6e59117b
a1434de441ca4077b26c9696b06a4cea 12e4f437b56a4badb839b816d6848d0a -
default default] [instance: eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4]
Instance failed to spawn: json.decoder.JSONDecodeError: Expecting
value: line 1 column 1 (char 0)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] Traceback (most recent call last):
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2368,
in _build_resources
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] yield resources
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2132,
in _build_and_run_instance
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] block_device_info=block_device_info)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line
3064, in spawn
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] block_device_info=block_device_info)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line
3442, in _create_image
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] fallback_from_host)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line
3533, in _create_and_inject_local_root
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] instance, size,
fallback_from_host)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line
7774, in _try_fetch_image_cache
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4]
trusted_certs=instance.trusted_certs)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py",
line 244, in cache
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] *args, **kwargs)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py",
line 890, in create_image
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] prepare_template(target=base,
*args, **kwargs)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line
274, in inner
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] return f(*args, **kwargs)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py",
line 240, in fetch_func_sync
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] fetch_func(target=target, *args,
**kwargs)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line
3527, in clone_fallback_to_fetch
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] libvirt_utils.fetch_image(*args,
**kwargs)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/utils.py", line 424,
in fetch_image
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] images.fetch_to_raw(context,
image_id, target, trusted_certs)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/images.py", line 145, in
fetch_to_raw
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] data = qemu_img_info(path_tmp)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/nova/virt/images.py", line 54, in
qemu_img_info
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] return
imageutils.QemuImgInfo(info, format='json')
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3/dist-packages/oslo_utils/imageutils.py", line 52, in
__init__
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] details =
json.loads(cmd_output or '{}')
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3.7/json/__init__.py", line 348, in loads
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] return
_default_decoder.decode(s)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3.7/json/decoder.py", line 337, in decode
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] obj, end =
self.raw_decode(s, idx=_w(s, 0).end())
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] File
"/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] raise
JSONDecodeError("Expecting value", s, err.value) from None
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] json.decoder.JSONDecodeError:
Expecting value: line 1 column 1 (char 0)
2023-03-14 09:48:23.734 8685 ERROR nova.compute.manager [instance:
eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4]
2023-03-14 09:48:23.738 8685 INFO nova.compute.manager
[req-aa77422c-3d55-43b9-b819-690c6e59117b
a1434de441ca4077b26c9696b06a4cea 12e4f437b56a4badb839b816d6848d0a -
default default] [instance: eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4]
Terminating instance
2023-03-14 09:48:23.746 8685 INFO nova.virt.libvirt.driver [-]
[instance: eb2b59f3-7bbd-4e86-a8d3-a0b7957959a4] Instance destroyed
successfully.
The instance was created using a Glance image, using RBD backed
ephemeral storage. An important thing to note is that the Glance image
service (and registry) is running in a different zone, with a different
Ceph cluster. When the image can be cloned, given the same zone and same
Ceph cluster, all still seems to work well.
Since it seems like something is going wrong with qemu_img_info() I
tried various versions of QEMU/KVM, being 1:3.1+dfsg-8+deb10u8,
1:3.1+dfsg-8+deb10u9 and 1:5.2+dfsg-9~bpo10+1. Alas, it doesn't help.
Tried different Glance backend types as well (RBD, file, Swift) but no
difference.
Downgrading using
{nova-common,nova-compute,nova-compute-kvm,python3-nova}=2:18.1.0-6
helps returning to a functional situation. Since it's gone from mirrors,
I wasn't able to try version 2:18.1.0-6+deb10u1. Obviously willing to do
that, if the packages are to be found somewhere.
Please advice what to do.
Thanks!
Cheers,
Kees
