Public bug reported:
Example hits from the last 14 days:
❯ logsearch log --project openstack/nova --job nova-ceph-multistore
--result FAILURE --days 14 "The image cannot be deleted because it is
in use through the backend store outside of Glance"
Builds with matching logs 5/15:
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| uuid | finished | pipeline | review
| branch |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| 5f7dccf493cf4b9d87d87a8f0a052192 | 2025-04-27T12:37:50 | check |
https://review.opendev.org/947212 | master |
| c7f28ff023b747849862c96817ffccdb | 2025-04-26T12:36:32 | check |
https://review.opendev.org/947212 | master |
| f94c9e6667874556849f625a01964367 | 2025-04-25T14:16:05 | check |
https://review.opendev.org/922496 | master |
| f169e50cabbf4e65a89a922b6ac9b29e | 2025-04-25T13:08:02 | check |
https://review.opendev.org/947966 | master |
| 0e53881b2e694ff8996b55bf7c477e79 | 2025-04-25T08:27:00 | check |
https://review.opendev.org/947212 | master |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
I only see it in the nova-ceph-multistore job but in various test cases:
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5f7/openstack/5f7dccf493cf4b9d87d87a8f0a052192/testr_results.html
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f94/openstack/f94c9e6667874556849f625a01964367/testr_results.html
*
https://40d1580bb656fd0ed240-3f272db0dacf207a646e9867f60c7e03.ssl.cf1.rackcdn.com/openstack/f169e50cabbf4e65a89a922b6ac9b29e/testr_results.html
I looked a bit into the occurrence
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/logs
below.
glance API logs:
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/log/controller/logs/screen-g-api.txt#12123-12132
Apr 27 11:52:19.736399 np0040547480 glance-api[98085]: DEBUG oslo_policy.policy
[None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="context_is_admin" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"auth_token": "***", "domain":
null, "global_request_id": null, "is_admin": false, "is_admin_project": true,
"project": "495c6985d5064e17970e2b26b58e56c6", "project_domain": "default",
"project_id": "495c6985d5064e17970e2b26b58e56c6", "read_only": false,
"request_id": "req-20c620eb-2315-4620-b145-98df5a0ed217", "resource_uuid":
null, "roles": ["reader", "member"], "service_catalog": [{"endpoints":
[{"publicURL": "https://158.69.64.188/identity";, "region": "RegionOne"}],
"name": "keystone", "type": "identity"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2/495c6985d5064e17970e2b26b58e56c6";, "region":
"RegionOne"}], "name": "nova_legacy", "type": "compute_legacy"}, {"endpoints":
[{"publicURL": "https://158.69.64.188/volume/v3";, "region": "RegionOne"}],
"name": "cinder", "type": "block-storage"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/placement";, "region": "RegionOne"}], "name":
"placement", "type": "placement"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/image";, "region": "RegionOne"}], "name": "glance",
"type": "image"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2.1";, "region": "RegionOne"}], "name": "nova",
"type": "compute"}, {"endpoints": [{"publicURL":
"https://158.69.64.188:9696/networking";, "region": "RegionOne"}], "name":
"neutron", "type": "network"}], "show_deleted": false, "system_scope": null,
"user": "ec6739598d2540019167fc2432e4d312", "user_domain": "default",
"user_identity": "ec6739598d2540019167fc2432e4d312
495c6985d5064e17970e2b26b58e56c6 - default default"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.748380 np0040547480 glance-api[98085]: DEBUG
glance.common.utils [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Sorted locations:
[{'store_name': 'robust'}] {{(pid=98085) sort_image_locations
/opt/stack/glance/glance/common/utils.py:735}}
Apr 27 11:52:19.750448 np0040547480 glance-api[98085]: DEBUG oslo_policy.policy
[None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="delete_image" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"backup_type": "daily",
"base_image_ref": "08f6a6f3-85eb-46c1-a2ab-e260218b0429", "boot_roles":
"reader,member", "checksum": null, "container_format": "bare", "created_at":
"2025-04-27T11:52:11.000000", "disk_format": "raw", "extra_properties": {},
"hw_cdrom_bus": "ide", "hw_disk_bus": "virtio", "hw_machine_type": "pc",
"hw_video_model": "virtio", "hw_vif_model": "virtio", "image_id":
"345b1922-5189-4d6f-a7af-bf28c7226051", "image_location": "snapshot",
"image_state": "available", "image_type": "backup", "instance_uuid":
"79f599ec-27d2-47cf-9072-be2ab65bee1b", "member": null, "min_disk": 1,
"min_ram": 0, "name": "tempest-backup-3-804999709", "os_glance_import_task":
"a39f3c07-8a30-48c3-ba47-5464019146e0", "os_hash_algo": "sha512",
"os_hash_value": null, "os_hidden": false, "owner":
"495c6985d5064e17970e2b26b58e56c6", "owner_id":
"495c6985d5064e17970e2b26b58e56c6", "owner_project_name":
"tempest-ServerActionsTestOtherB-1525757365", "owner_specified.openstack.md5":
"", "owner_specified.openstack.object": "images/cirros-raw",
"owner_specified.openstack.sha256": "", "owner_user_name":
"tempest-ServerActionsTestOtherB-1525757365-project-member", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "protected": false, "size": 1073741824,
"status": "active", "tags": [], "updated_at": "2025-04-27T11:52:18.000000",
"user_id": "ec6739598d2540019167fc2432e4d312", "virtual_size": null,
"visibility": "private"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.751720 np0040547480 glance-api[98085]: DEBUG
glance_store.multi_backend [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Attempting to import
store rbd {{(pid=98085) _load_multi_store
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/multi_backend.py:170}}
Apr 27 11:52:19.773854 np0040547480 glance-api[98085]: DEBUG
glance_store.capabilities [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Store
glance_store._drivers.rbd.Store doesn't support updating dynamic storage
capabilities. Please overwrite 'update_capabilities' method of the store to
implement updating logics if needed. {{(pid=98085) update_capabilities
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/capabilities.py:91}}
Apr 27 11:52:19.774138 np0040547480 glance-api[98085]: DEBUG
glance_store.driver [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Late loading
location class glance_store._drivers.rbd.StoreLocation {{(pid=98085)
get_store_location_class
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/driver.py:133}}
Apr 27 11:52:19.774268 np0040547480 glance-api[98085]: DEBUG
glance_store.location [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Registering scheme
rbd with {'robust': {'store': <glance_store._drivers.rbd.Store object at
0x7687f29b94c0>, 'location_class': <class
'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'rbd'}}
{{(pid=98085) register_scheme_backend_map
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/location.py:132}}
Apr 27 11:52:20.682141 np0040547480 glance-api[98085]: WARNING
glance_store._drivers.rbd [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Remove image
345b1922-5189-4d6f-a7af-bf28c7226051 failed. It is in use.: rbd.ImageBusy:
[errno 16] RBD image is busy (error removing image)
Apr 27 11:52:20.687798 np0040547480 glance-api[98085]: WARNING
glance.api.v2.images [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Image
345b1922-5189-4d6f-a7af-bf28c7226051 could not be deleted because it is in use:
The image cannot be deleted because it is in use through the backend store
outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be
deleted because it is in use through the backend store outside of Glance.
Apr 27 11:52:20.689146 np0040547480 glance-api[98085]: INFO
eventlet.wsgi.server [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member]
158.69.64.188,158.69.64.188,158.69.64.188 - - [27/Apr/2025 11:52:20] "DELETE
/v2/images/345b1922-5189-4d6f-a7af-bf28c7226051 HTTP/1.1" 409 462 0.956102
** Affects: glance
Importance: Undecided
Status: New
** Tags: ceph gate multistore tempest
** Description changed:
Example hits from the last 14 days:
❯ logsearch log --project openstack/nova --job nova-ceph-multistore
--result FAILURE --days 14 "The image cannot be deleted because it is
in use through the backend store outside of Glance"
- ─╯
-
Builds with matching logs 5/15:
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| uuid | finished | pipeline | review
| branch |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| 5f7dccf493cf4b9d87d87a8f0a052192 | 2025-04-27T12:37:50 | check |
https://review.opendev.org/947212 | master |
| c7f28ff023b747849862c96817ffccdb | 2025-04-26T12:36:32 | check |
https://review.opendev.org/947212 | master |
| f94c9e6667874556849f625a01964367 | 2025-04-25T14:16:05 | check |
https://review.opendev.org/922496 | master |
| f169e50cabbf4e65a89a922b6ac9b29e | 2025-04-25T13:08:02 | check |
https://review.opendev.org/947966 | master |
| 0e53881b2e694ff8996b55bf7c477e79 | 2025-04-25T08:27:00 | check |
https://review.opendev.org/947212 | master |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
-
I only see it in the nova-ceph-multistore job but in various test cases:
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5f7/openstack/5f7dccf493cf4b9d87d87a8f0a052192/testr_results.html
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f94/openstack/f94c9e6667874556849f625a01964367/testr_results.html
*
https://40d1580bb656fd0ed240-3f272db0dacf207a646e9867f60c7e03.ssl.cf1.rackcdn.com/openstack/f169e50cabbf4e65a89a922b6ac9b29e/testr_results.html
I looked a bit into the occurrence
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/logs
below.
-
glance API logs:
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/log/controller/logs/screen-g-api.txt#12123-12132
Apr 27 11:52:19.736399 np0040547480 glance-api[98085]: DEBUG
oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="context_is_admin" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"auth_token": "***", "domain":
null, "global_request_id": null, "is_admin": false, "is_admin_project": true,
"project": "495c6985d5064e17970e2b26b58e56c6", "project_domain": "default",
"project_id": "495c6985d5064e17970e2b26b58e56c6", "read_only": false,
"request_id": "req-20c620eb-2315-4620-b145-98df5a0ed217", "resource_uuid":
null, "roles": ["reader", "member"], "service_catalog": [{"endpoints":
[{"publicURL": "https://158.69.64.188/identity";, "region": "RegionOne"}],
"name": "keystone", "type": "identity"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2/495c6985d5064e17970e2b26b58e56c6";, "region":
"RegionOne"}], "name": "nova_legacy", "type": "compute_legacy"}, {"endpoints":
[{"publicURL": "https://158.69.64.188/volume/v3";, "region": "RegionOne"}],
"name": "cinder", "type": "block-storage"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/placement";, "region": "RegionOne"}], "name":
"placement", "type": "placement"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/image";, "region": "RegionOne"}], "name": "glance",
"type": "image"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2.1";, "region": "RegionOne"}], "name": "nova",
"type": "compute"}, {"endpoints": [{"publicURL":
"https://158.69.64.188:9696/networking";, "region": "RegionOne"}], "name":
"neutron", "type": "network"}], "show_deleted": false, "system_scope": null,
"user": "ec6739598d2540019167fc2432e4d312", "user_domain": "default",
"user_identity": "ec6739598d2540019167fc2432e4d312
495c6985d5064e17970e2b26b58e56c6 - default default"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.748380 np0040547480 glance-api[98085]: DEBUG
glance.common.utils [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Sorted locations:
[{'store_name': 'robust'}] {{(pid=98085) sort_image_locations
/opt/stack/glance/glance/common/utils.py:735}}
Apr 27 11:52:19.750448 np0040547480 glance-api[98085]: DEBUG
oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="delete_image" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"backup_type": "daily",
"base_image_ref": "08f6a6f3-85eb-46c1-a2ab-e260218b0429", "boot_roles":
"reader,member", "checksum": null, "container_format": "bare", "created_at":
"2025-04-27T11:52:11.000000", "disk_format": "raw", "extra_properties": {},
"hw_cdrom_bus": "ide", "hw_disk_bus": "virtio", "hw_machine_type": "pc",
"hw_video_model": "virtio", "hw_vif_model": "virtio", "image_id":
"345b1922-5189-4d6f-a7af-bf28c7226051", "image_location": "snapshot",
"image_state": "available", "image_type": "backup", "instance_uuid":
"79f599ec-27d2-47cf-9072-be2ab65bee1b", "member": null, "min_disk": 1,
"min_ram": 0, "name": "tempest-backup-3-804999709", "os_glance_import_task":
"a39f3c07-8a30-48c3-ba47-5464019146e0", "os_hash_algo": "sha512",
"os_hash_value": null, "os_hidden": false, "owner":
"495c6985d5064e17970e2b26b58e56c6", "owner_id":
"495c6985d5064e17970e2b26b58e56c6", "owner_project_name":
"tempest-ServerActionsTestOtherB-1525757365", "owner_specified.openstack.md5":
"", "owner_specified.openstack.object": "images/cirros-raw",
"owner_specified.openstack.sha256": "", "owner_user_name":
"tempest-ServerActionsTestOtherB-1525757365-project-member", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "protected": false, "size": 1073741824,
"status": "active", "tags": [], "updated_at": "2025-04-27T11:52:18.000000",
"user_id": "ec6739598d2540019167fc2432e4d312", "virtual_size": null,
"visibility": "private"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.751720 np0040547480 glance-api[98085]: DEBUG
glance_store.multi_backend [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Attempting to import
store rbd {{(pid=98085) _load_multi_store
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/multi_backend.py:170}}
Apr 27 11:52:19.773854 np0040547480 glance-api[98085]: DEBUG
glance_store.capabilities [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Store
glance_store._drivers.rbd.Store doesn't support updating dynamic storage
capabilities. Please overwrite 'update_capabilities' method of the store to
implement updating logics if needed. {{(pid=98085) update_capabilities
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/capabilities.py:91}}
Apr 27 11:52:19.774138 np0040547480 glance-api[98085]: DEBUG
glance_store.driver [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Late loading
location class glance_store._drivers.rbd.StoreLocation {{(pid=98085)
get_store_location_class
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/driver.py:133}}
Apr 27 11:52:19.774268 np0040547480 glance-api[98085]: DEBUG
glance_store.location [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Registering scheme
rbd with {'robust': {'store': <glance_store._drivers.rbd.Store object at
0x7687f29b94c0>, 'location_class': <class
'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'rbd'}}
{{(pid=98085) register_scheme_backend_map
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/location.py:132}}
Apr 27 11:52:20.682141 np0040547480 glance-api[98085]: WARNING
glance_store._drivers.rbd [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Remove image
345b1922-5189-4d6f-a7af-bf28c7226051 failed. It is in use.: rbd.ImageBusy:
[errno 16] RBD image is busy (error removing image)
Apr 27 11:52:20.687798 np0040547480 glance-api[98085]: WARNING
glance.api.v2.images [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Image
345b1922-5189-4d6f-a7af-bf28c7226051 could not be deleted because it is in use:
The image cannot be deleted because it is in use through the backend store
outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be
deleted because it is in use through the backend store outside of Glance.
Apr 27 11:52:20.689146 np0040547480 glance-api[98085]: INFO
eventlet.wsgi.server [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member]
158.69.64.188,158.69.64.188,158.69.64.188 - - [27/Apr/2025 11:52:20] "DELETE
/v2/images/345b1922-5189-4d6f-a7af-bf28c7226051 HTTP/1.1" 409 462 0.956102
** Tags added: ceph gate multistore tempest
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2109428
Title:
Image delete fails in tempest nova-ceph-multistore job with: "The
image cannot be deleted because it is in use through the backend store
outside of Glance"
Status in Glance:
New
Bug description:
Example hits from the last 14 days:
❯ logsearch log --project openstack/nova --job nova-ceph-multistore
--result FAILURE --days 14 "The image cannot be deleted because it is
in use through the backend store outside of Glance"
Builds with matching logs 5/15:
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| uuid | finished | pipeline | review
| branch |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| 5f7dccf493cf4b9d87d87a8f0a052192 | 2025-04-27T12:37:50 | check |
https://review.opendev.org/947212 | master |
| c7f28ff023b747849862c96817ffccdb | 2025-04-26T12:36:32 | check |
https://review.opendev.org/947212 | master |
| f94c9e6667874556849f625a01964367 | 2025-04-25T14:16:05 | check |
https://review.opendev.org/922496 | master |
| f169e50cabbf4e65a89a922b6ac9b29e | 2025-04-25T13:08:02 | check |
https://review.opendev.org/947966 | master |
| 0e53881b2e694ff8996b55bf7c477e79 | 2025-04-25T08:27:00 | check |
https://review.opendev.org/947212 | master |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
I only see it in the nova-ceph-multistore job but in various test cases:
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5f7/openstack/5f7dccf493cf4b9d87d87a8f0a052192/testr_results.html
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f94/openstack/f94c9e6667874556849f625a01964367/testr_results.html
*
https://40d1580bb656fd0ed240-3f272db0dacf207a646e9867f60c7e03.ssl.cf1.rackcdn.com/openstack/f169e50cabbf4e65a89a922b6ac9b29e/testr_results.html
I looked a bit into the occurrence
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/logs
below.
glance API logs:
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/log/controller/logs/screen-g-api.txt#12123-12132
Apr 27 11:52:19.736399 np0040547480 glance-api[98085]: DEBUG
oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="context_is_admin" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"auth_token": "***", "domain":
null, "global_request_id": null, "is_admin": false, "is_admin_project": true,
"project": "495c6985d5064e17970e2b26b58e56c6", "project_domain": "default",
"project_id": "495c6985d5064e17970e2b26b58e56c6", "read_only": false,
"request_id": "req-20c620eb-2315-4620-b145-98df5a0ed217", "resource_uuid":
null, "roles": ["reader", "member"], "service_catalog": [{"endpoints":
[{"publicURL": "https://158.69.64.188/identity";, "region": "RegionOne"}],
"name": "keystone", "type": "identity"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2/495c6985d5064e17970e2b26b58e56c6";, "region":
"RegionOne"}], "name": "nova_legacy", "type": "compute_legacy"}, {"endpoints":
[{"publicURL": "https://158.69.64.188/volume/v3";, "region": "RegionOne"}],
"name": "cinder", "type": "block-storage"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/placement";, "region": "RegionOne"}], "name":
"placement", "type": "placement"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/image";, "region": "RegionOne"}], "name": "glance",
"type": "image"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2.1";, "region": "RegionOne"}], "name": "nova",
"type": "compute"}, {"endpoints": [{"publicURL":
"https://158.69.64.188:9696/networking";, "region": "RegionOne"}], "name":
"neutron", "type": "network"}], "show_deleted": false, "system_scope": null,
"user": "ec6739598d2540019167fc2432e4d312", "user_domain": "default",
"user_identity": "ec6739598d2540019167fc2432e4d312
495c6985d5064e17970e2b26b58e56c6 - default default"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.748380 np0040547480 glance-api[98085]: DEBUG
glance.common.utils [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Sorted locations:
[{'store_name': 'robust'}] {{(pid=98085) sort_image_locations
/opt/stack/glance/glance/common/utils.py:735}}
Apr 27 11:52:19.750448 np0040547480 glance-api[98085]: DEBUG
oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="delete_image" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"backup_type": "daily",
"base_image_ref": "08f6a6f3-85eb-46c1-a2ab-e260218b0429", "boot_roles":
"reader,member", "checksum": null, "container_format": "bare", "created_at":
"2025-04-27T11:52:11.000000", "disk_format": "raw", "extra_properties": {},
"hw_cdrom_bus": "ide", "hw_disk_bus": "virtio", "hw_machine_type": "pc",
"hw_video_model": "virtio", "hw_vif_model": "virtio", "image_id":
"345b1922-5189-4d6f-a7af-bf28c7226051", "image_location": "snapshot",
"image_state": "available", "image_type": "backup", "instance_uuid":
"79f599ec-27d2-47cf-9072-be2ab65bee1b", "member": null, "min_disk": 1,
"min_ram": 0, "name": "tempest-backup-3-804999709", "os_glance_import_task":
"a39f3c07-8a30-48c3-ba47-5464019146e0", "os_hash_algo": "sha512",
"os_hash_value": null, "os_hidden": false, "owner":
"495c6985d5064e17970e2b26b58e56c6", "owner_id":
"495c6985d5064e17970e2b26b58e56c6", "owner_project_name":
"tempest-ServerActionsTestOtherB-1525757365", "owner_specified.openstack.md5":
"", "owner_specified.openstack.object": "images/cirros-raw",
"owner_specified.openstack.sha256": "", "owner_user_name":
"tempest-ServerActionsTestOtherB-1525757365-project-member", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "protected": false, "size": 1073741824,
"status": "active", "tags": [], "updated_at": "2025-04-27T11:52:18.000000",
"user_id": "ec6739598d2540019167fc2432e4d312", "virtual_size": null,
"visibility": "private"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.751720 np0040547480 glance-api[98085]: DEBUG
glance_store.multi_backend [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Attempting to import
store rbd {{(pid=98085) _load_multi_store
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/multi_backend.py:170}}
Apr 27 11:52:19.773854 np0040547480 glance-api[98085]: DEBUG
glance_store.capabilities [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Store
glance_store._drivers.rbd.Store doesn't support updating dynamic storage
capabilities. Please overwrite 'update_capabilities' method of the store to
implement updating logics if needed. {{(pid=98085) update_capabilities
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/capabilities.py:91}}
Apr 27 11:52:19.774138 np0040547480 glance-api[98085]: DEBUG
glance_store.driver [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Late loading
location class glance_store._drivers.rbd.StoreLocation {{(pid=98085)
get_store_location_class
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/driver.py:133}}
Apr 27 11:52:19.774268 np0040547480 glance-api[98085]: DEBUG
glance_store.location [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Registering scheme
rbd with {'robust': {'store': <glance_store._drivers.rbd.Store object at
0x7687f29b94c0>, 'location_class': <class
'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'rbd'}}
{{(pid=98085) register_scheme_backend_map
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/location.py:132}}
Apr 27 11:52:20.682141 np0040547480 glance-api[98085]: WARNING
glance_store._drivers.rbd [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Remove image
345b1922-5189-4d6f-a7af-bf28c7226051 failed. It is in use.: rbd.ImageBusy:
[errno 16] RBD image is busy (error removing image)
Apr 27 11:52:20.687798 np0040547480 glance-api[98085]: WARNING
glance.api.v2.images [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Image
345b1922-5189-4d6f-a7af-bf28c7226051 could not be deleted because it is in use:
The image cannot be deleted because it is in use through the backend store
outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be
deleted because it is in use through the backend store outside of Glance.
Apr 27 11:52:20.689146 np0040547480 glance-api[98085]: INFO
eventlet.wsgi.server [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member]
158.69.64.188,158.69.64.188,158.69.64.188 - - [27/Apr/2025 11:52:20] "DELETE
/v2/images/345b1922-5189-4d6f-a7af-bf28c7226051 HTTP/1.1" 409 462 0.956102
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2109428/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
