Public bug reported: Description =========== Having upgraded three OpenStack deployments to Victoria, we have noticed that snapshots are now failing to be created. When a user attempts this via Horizon they receive an error such as:
Error: Unable to create snapshot. Details Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'AttributeError'> (HTTP 500) (Request-ID: req-7e4b0824-9884-4d50 -b8ee-5184e245a11d) It appears that this can be worked around by adding 'from eventlet import tpool' and adjusting the 'eventlet.tpool.execute' call to match in nova/utils.py Steps to reproduce ================== Via Horizon on a Victoria deployment: - Create an instance - Suspend the instance - Create a snapshot Expected result =============== A snapshot should have been created Actual result ============= Error message as above Environment =========== 1. Exact version of OpenStack you are running. Git hash cb4963ba8c4babb7e460d71daf91099fc6da1dc6 2. Which hypervisor did you use? Libvirt+KVM 2. Which storage type did you use? Ceph Octopus 3. Which networking type did you use? Neutron with Linux Bridge Logs & Configs ============== The full traceback from the API is as follows: Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] Unexpected exception in API method: AttributeError: module 'eventlet' has no attribute 'tpool' 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 1244, in _action_create_image 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi metadata) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 156, in inner 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 3374, in snapshot_volume_backed 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return self.image_api.create(context, image_meta) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 1224, in create 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return session.create(context, image_info, data=data) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 555, in create 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi sharing_member_id=sharing_member_id) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 670, in _create_v2 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi image = self._upload_data(context, image_id, data) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 591, in _upload_data 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi args=(image_id, data)) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/utils.py", line 694, in tpool_execute 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi eventlet.tpool.execute(func, *args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi AttributeError: module 'eventlet' has no attribute 'tpool' 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.079 26063 INFO nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'AttributeError'> ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1915400 Title: Snapshots fail with traceback from API Status in OpenStack Compute (nova): New Bug description: Description =========== Having upgraded three OpenStack deployments to Victoria, we have noticed that snapshots are now failing to be created. When a user attempts this via Horizon they receive an error such as: Error: Unable to create snapshot. Details Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'AttributeError'> (HTTP 500) (Request-ID: req- 7e4b0824-9884-4d50-b8ee-5184e245a11d) It appears that this can be worked around by adding 'from eventlet import tpool' and adjusting the 'eventlet.tpool.execute' call to match in nova/utils.py Steps to reproduce ================== Via Horizon on a Victoria deployment: - Create an instance - Suspend the instance - Create a snapshot Expected result =============== A snapshot should have been created Actual result ============= Error message as above Environment =========== 1. Exact version of OpenStack you are running. Git hash cb4963ba8c4babb7e460d71daf91099fc6da1dc6 2. Which hypervisor did you use? Libvirt+KVM 2. Which storage type did you use? Ceph Octopus 3. Which networking type did you use? Neutron with Linux Bridge Logs & Configs ============== The full traceback from the API is as follows: Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] Unexpected exception in API method: AttributeError: module 'eventlet' has no attribute 'tpool' 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 1244, in _action_create_image 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi metadata) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 156, in inner 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 3374, in snapshot_volume_backed 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return self.image_api.create(context, image_meta) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 1224, in create 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return session.create(context, image_info, data=data) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 555, in create 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi sharing_member_id=sharing_member_id) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 670, in _create_v2 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi image = self._upload_data(context, image_id, data) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 591, in _upload_data 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi args=(image_id, data)) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/utils.py", line 694, in tpool_execute 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi eventlet.tpool.execute(func, *args, **kwargs) 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi AttributeError: module 'eventlet' has no attribute 'tpool' 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.079 26063 INFO nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'AttributeError'> To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1915400/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
