Public bug reported: Description ===========
When an ephemeral instance (image booted from image) is shelved, Nova writes the root volume of the instance to a temporary Glance image to make sure it persists while the instance is shelved. Upon unshelve the volume is written back out to disk, and the temporary image is removed. Upon unshelving the instance, when the libvirt domain metadata for the instance is generated the temporary image's ID is written to the metadata, instead of the ID of the original image the instance is based on like it should. This is is corrected when the instance is cold restarted, resized or cold migrated but the incorrect image ID can persist if the state of the instance does not change following the unshelve, which causes issues for other services such as Ceilometer which rely on this data being correct. Steps to reproduce ================== 1. Launch a new ephemeral instance, booted from image. 2. Confirm the correct image ID of the active ephemeral instance: $ openstack server show <instance> 3. Check the libvirt domain XML (the image ID should be correct): $ virsh dumpxml instance-xxxxxxxx 4. Shelve the instance: $ openstack server shelve <instance> 5. Unshelve the instance: $ openstack server unshelve <instance> 6. Check the libvirt domain XML again (the image ID will be incorrect): $ virsh dumpxml instance-xxxxxxxx Expected result =============== Given the following list of images (taken while the instance 'test2' is shelved): $ openstack image list +--------------------------------------+--------------------------------------+--------+ | ID | Name | Status | +--------------------------------------+--------------------------------------+--------+ | 00000000-0000-0000-0000-000000000001 | alt_tempest_cirros-0.3.4-x86_64 | active | | 4c1e8dbe-7ef5-44be-8b3c-e36f72d93589 | cirros-0.6.2-x86_64 | active | | a734e8cf-376d-4d00-a51e-b57995a4fc24 | octavia_image | active | | 00000000-0000-0000-0000-000000000000 | tempest_cirros-0.3.4-x86_64 | active | | f8e04922-c715-4ffe-82ef-cddace05f331 | test2-shelved | active | | e69daa32-b950-40d6-bc5e-599e73f5c838 | trove-victoria-ubuntu-focal-20240514 | active | +--------------------------------------+--------------------------------------+--------+ libvirt domain metadata before unshelve: $ sudo virsh dumpxml 7a8af73e-9aa2-4f5d-bb20-7f5623adcd58 | xpath -q -e /domain/metadata <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> <nova:package version="17.0.14" /> <nova:name>test2</nova:name> <nova:creationTime>2025-02-27 01:26:17</nova:creationTime> <nova:flavor name="c1.c1r05"> <nova:memory>512</nova:memory> <nova:disk>5</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> </nova:owner> <nova:root type="image" uuid="4c1e8dbe-7ef5-44be-8b3c-e36f72d93589" /> <!-- cirros-0.6.2-x86_64 --> </nova:instance> </metadata> After unshelve, the image ID should stay the same. Actual result ============= After unshelve: $ sudo virsh dumpxml 7a8af73e-9aa2-4f5d-bb20-7f5623adcd58 | xpath -q -e /domain/metadata <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> <nova:package version="17.0.14" /> <nova:name>test2</nova:name> <nova:creationTime>2025-02-27 01:17:00</nova:creationTime> <nova:flavor name="c1.c1r05"> <nova:memory>512</nova:memory> <nova:disk>5</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> </nova:owner> <nova:root type="image" uuid="f8e04922-c715-4ffe-82ef-cddace05f331" /> <!-- test2-shelved --> </nova:instance> </metadata> Environment =========== We have experienced it on Nova Queens through to Train, but looking at the implementation all Nova versions up to master are affected. ** Affects: nova Importance: Undecided Status: New ** Description changed: Description =========== When an ephemeral instance (image booted from image) is shelved, Nova writes the root volume of the instance to a temporary Glance image to make sure it persists while the instance is shelved. Upon unshelve the volume is written back out to disk, and the temporary image is removed. Upon unshelving the instance, when the libvirt domain metadata for the instance is generated the temporary image's ID is written to the metadata, instead of the ID of the original image the instance is based on like it should. This is is corrected when the instance is cold restarted, resized or cold migrated but the incorrect image ID can persist if the state of the instance does not change following the unshelve, which causes issues for other services such as Ceilometer which rely on this data being correct. - Steps to reproduce ================== 1. Launch a new ephemeral instance, booted from image. 2. Confirm the correct image ID of the active ephemeral instance: $ openstack server show <instance> 3. Check the libvirt domain XML (the image ID should be correct): $ virsh dumpxml instance-xxxxxxxx 4. Shelve the instance: $ openstack server shelve <instance> 5. Unshelve the instance: $ openstack server unshelve <instance> 6. Check the libvirt domain XML again (the image ID will be incorrect): $ virsh dumpxml instance-xxxxxxxx - Expected result =============== Given the following list of images (taken while the instance 'test2' is shelved): $ openstack image list +--------------------------------------+--------------------------------------+--------+ | ID | Name | Status | +--------------------------------------+--------------------------------------+--------+ | 00000000-0000-0000-0000-000000000001 | alt_tempest_cirros-0.3.4-x86_64 | active | | 4c1e8dbe-7ef5-44be-8b3c-e36f72d93589 | cirros-0.6.2-x86_64 | active | | a734e8cf-376d-4d00-a51e-b57995a4fc24 | octavia_image | active | | 00000000-0000-0000-0000-000000000000 | tempest_cirros-0.3.4-x86_64 | active | | f8e04922-c715-4ffe-82ef-cddace05f331 | test2-shelved | active | | e69daa32-b950-40d6-bc5e-599e73f5c838 | trove-victoria-ubuntu-focal-20240514 | active | +--------------------------------------+--------------------------------------+--------+ libvirt domain metadata before unshelve: + $ sudo virsh dumpxml 7a8af73e-9aa2-4f5d-bb20-7f5623adcd58 | xpath -q -e /domain/metadata <metadata> - <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> - <nova:package version="17.0.14" /> - <nova:name>test2</nova:name> - <nova:creationTime>2025-02-27 01:26:17</nova:creationTime> - <nova:flavor name="c1.c1r05"> - <nova:memory>512</nova:memory> - <nova:disk>5</nova:disk> - <nova:swap>0</nova:swap> - <nova:ephemeral>0</nova:ephemeral> - <nova:vcpus>1</nova:vcpus> - </nova:flavor> - <nova:owner> - <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> - <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> - </nova:owner> - <nova:root type="image" uuid="4c1e8dbe-7ef5-44be-8b3c-e36f72d93589" /> <!-- cirros-0.6.2-x86_64 --> - </nova:instance> - </metadata> + <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> + <nova:package version="17.0.14" /> + <nova:name>test2</nova:name> + <nova:creationTime>2025-02-27 01:26:17</nova:creationTime> + <nova:flavor name="c1.c1r05"> + <nova:memory>512</nova:memory> + <nova:disk>5</nova:disk> + <nova:swap>0</nova:swap> + <nova:ephemeral>0</nova:ephemeral> + <nova:vcpus>1</nova:vcpus> + </nova:flavor> + <nova:owner> + <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> + <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> + </nova:owner> + <nova:root type="image" uuid="4c1e8dbe-7ef5-44be-8b3c-e36f72d93589" /> <!-- cirros-0.6.2-x86_64 --> + </nova:instance> + </metadata> After unshelve, the image ID should stay the same. - Actual result ============= After unshelve: $ sudo virsh dumpxml 7a8af73e-9aa2-4f5d-bb20-7f5623adcd58 | xpath -q -e /domain/metadata <metadata> - <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> - <nova:package version="17.0.14" /> - <nova:name>test2</nova:name> - <nova:creationTime>2025-02-27 01:17:00</nova:creationTime> - <nova:flavor name="c1.c1r05"> - <nova:memory>512</nova:memory> - <nova:disk>5</nova:disk> - <nova:swap>0</nova:swap> - <nova:ephemeral>0</nova:ephemeral> - <nova:vcpus>1</nova:vcpus> - </nova:flavor> - <nova:owner> - <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> - <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> - </nova:owner> - <nova:root type="image" uuid="f8e04922-c715-4ffe-82ef-cddace05f331" /> <!-- test2-shelved --> - </nova:instance> - </metadata> - + <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> + <nova:package version="17.0.14" /> + <nova:name>test2</nova:name> + <nova:creationTime>2025-02-27 01:17:00</nova:creationTime> + <nova:flavor name="c1.c1r05"> + <nova:memory>512</nova:memory> + <nova:disk>5</nova:disk> + <nova:swap>0</nova:swap> + <nova:ephemeral>0</nova:ephemeral> + <nova:vcpus>1</nova:vcpus> + </nova:flavor> + <nova:owner> + <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> + <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> + </nova:owner> + <nova:root type="image" uuid="f8e04922-c715-4ffe-82ef-cddace05f331" /> <!-- test2-shelved --> + </nova:instance> + </metadata> Environment =========== We have experienced it on Nova Queens through to Train, but looking at the implementation all Nova versions up to master are affected. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2100588 Title: Incorrect image ID written to libvirt metadata when unshelving ephemeral instances Status in OpenStack Compute (nova): New Bug description: Description =========== When an ephemeral instance (image booted from image) is shelved, Nova writes the root volume of the instance to a temporary Glance image to make sure it persists while the instance is shelved. Upon unshelve the volume is written back out to disk, and the temporary image is removed. Upon unshelving the instance, when the libvirt domain metadata for the instance is generated the temporary image's ID is written to the metadata, instead of the ID of the original image the instance is based on like it should. This is is corrected when the instance is cold restarted, resized or cold migrated but the incorrect image ID can persist if the state of the instance does not change following the unshelve, which causes issues for other services such as Ceilometer which rely on this data being correct. Steps to reproduce ================== 1. Launch a new ephemeral instance, booted from image. 2. Confirm the correct image ID of the active ephemeral instance: $ openstack server show <instance> 3. Check the libvirt domain XML (the image ID should be correct): $ virsh dumpxml instance-xxxxxxxx 4. Shelve the instance: $ openstack server shelve <instance> 5. Unshelve the instance: $ openstack server unshelve <instance> 6. Check the libvirt domain XML again (the image ID will be incorrect): $ virsh dumpxml instance-xxxxxxxx Expected result =============== Given the following list of images (taken while the instance 'test2' is shelved): $ openstack image list +--------------------------------------+--------------------------------------+--------+ | ID | Name | Status | +--------------------------------------+--------------------------------------+--------+ | 00000000-0000-0000-0000-000000000001 | alt_tempest_cirros-0.3.4-x86_64 | active | | 4c1e8dbe-7ef5-44be-8b3c-e36f72d93589 | cirros-0.6.2-x86_64 | active | | a734e8cf-376d-4d00-a51e-b57995a4fc24 | octavia_image | active | | 00000000-0000-0000-0000-000000000000 | tempest_cirros-0.3.4-x86_64 | active | | f8e04922-c715-4ffe-82ef-cddace05f331 | test2-shelved | active | | e69daa32-b950-40d6-bc5e-599e73f5c838 | trove-victoria-ubuntu-focal-20240514 | active | +--------------------------------------+--------------------------------------+--------+ libvirt domain metadata before unshelve: $ sudo virsh dumpxml 7a8af73e-9aa2-4f5d-bb20-7f5623adcd58 | xpath -q -e /domain/metadata <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> <nova:package version="17.0.14" /> <nova:name>test2</nova:name> <nova:creationTime>2025-02-27 01:26:17</nova:creationTime> <nova:flavor name="c1.c1r05"> <nova:memory>512</nova:memory> <nova:disk>5</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> </nova:owner> <nova:root type="image" uuid="4c1e8dbe-7ef5-44be-8b3c-e36f72d93589" /> <!-- cirros-0.6.2-x86_64 --> </nova:instance> </metadata> After unshelve, the image ID should stay the same. Actual result ============= After unshelve: $ sudo virsh dumpxml 7a8af73e-9aa2-4f5d-bb20-7f5623adcd58 | xpath -q -e /domain/metadata <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";> <nova:package version="17.0.14" /> <nova:name>test2</nova:name> <nova:creationTime>2025-02-27 01:17:00</nova:creationTime> <nova:flavor name="c1.c1r05"> <nova:memory>512</nova:memory> <nova:disk>5</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="a55b109e1508472cb4466bc84786680e">jeremy</nova:user> <nova:project uuid="ab5d28cb6e2c4b7d89c8e0f96ac8604e">test1</nova:project> </nova:owner> <nova:root type="image" uuid="f8e04922-c715-4ffe-82ef-cddace05f331" /> <!-- test2-shelved --> </nova:instance> </metadata> Environment =========== We have experienced it on Nova Queens through to Train, but looking at the implementation all Nova versions up to master are affected. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2100588/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
