Reviewed: https://review.openstack.org/555093 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4 Submitter: Zuul Branch: master
commit b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4 Author: melanie witt <[email protected]> Date: Wed Mar 21 22:57:50 2018 +0000 Move _make_instance_list call outside of DB transaction context The _make_instance_list method is used to make an InstanceList object out of database dict-like instance objects. It's possible while making the list that the various _from_db_object methods that are called might do their own database writes. Currently, we're calling _make_instance_list nested inside of a 'reader' database transaction context and we hit the error: TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction during the _make_instance_list call if anything tries to do a database write. The scenario encountered was after an upgrade to Pike, older service records without UUIDs were attempted to be updated with UUIDs upon access, and that access happened to be during an instance list, so it failed when trying to write the service UUID while nested inside the 'reader' database transaction context. This simply moves the _make_instance_list method call out from the @db.select_db_reader_mode decorated _get_by_filters_impl method to the get_by_filters method to remove the nesting. Closes-Bug: #1746509 Change-Id: Ifadf408802cc15eb9769d2dc1fc920426bb7fc20 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1746509 Title: TypeError: Can't upgrade a READER transaction to a WRITER mid- transaction Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Compute (nova) queens series: In Progress Bug description: Hi, I was running OPenstack Newton with no nova_cell0 database and placement-api setup . After migrate to Openstack Pike and correctly setup the nova_cell0 and placement-api everything is working fine except the openstack server list on tenant that already exist . For example : 1. For a new tenant created after the migration at Pike. nova --os-project-name="New Project" list +--------------------------------------+-----------------+--------+------------+-------------+---------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------+--------+------------+-------------+---------------------------+ | c41c7e8d-4bc0-4a0f-a9d3-dc719ae2aff0 | SAMPLE_VM | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.8 | | 3d3d3e10-f326-4a92-9253-1511e738d1cc | SECOND_INSTANCE | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.5 | +--------------------------------------+-----------------+--------+------------+-------------+---------------------------+ 2. For a old tenant created before the migration at Newton . nova --os-project-name="InterCon" list ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-0dd4ef4d-54c2-4cfd-b8d9-636c5736ef5f) And here the log related to this error . 2018-01-31 07:45:35.832 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Searching by: {'deleted': False} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2311 2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282 2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Skipping already-collected cell0 list _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2503 2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Listing 1000 instances in cell ea0a32fb-c148-4d4f-a42d-4e46356ff926 _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2506 2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282 2018-01-31 07:45:35.934 2340 DEBUG nova.objects.service [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Generated UUID f9ae1e59-21be-4fc3-9b02-ebb4c35783ec for service 19 _from_db_object /usr/lib/python2.7/site-packages/nova/objects/service.py:245 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Unexpected exception in API method: TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 336, in wrapped 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 168, in detail 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 311, in _get_servers 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2415, in get_all 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_dirs=sort_dirs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2512, in _get_instances_by_filters_all_cells 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2543, in _get_instances_by_filters 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1245, in get_by_filters 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1235, in _get_by_filters_impl 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1198, in _make_instance_list 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 448, in _from_db_object 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions objects.Service, db_inst['services']) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 1121, in obj_make_list 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **extra_args) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 246, in _from_db_object 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions service.save() 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 363, in save 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions db_service = db.service_update(self._context, self.id, updates) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return IMPL.service_update(context, service_id, values) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self.force_reraise() 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions with ctxt_mgr.writer.using(context): 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return self.gen.next() 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1034, in _transaction_scope 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions context=context) as resource: 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 684, in _produce_block 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self._writer() 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 702, in _writer 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions "Can't upgrade a READER transaction " 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction 2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions 2018-01-31 07:45:35.938 2340 INFO nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.TypeError'> 2018-01-31 07:45:35.938 2340 DEBUG nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.TypeError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1029 2018-01-31 07:45:35.939 2340 INFO nova.osapi_compute.wsgi.server [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] 192.168.0.180 "GET /v2.1/servers/detail?all_tenants=True HTTP/1.1" status: 500 len: 626 time: 0.2633660 but at same tenant I can show a instance information . nova --os-project-name="Intercon" show 5249185f-0a8e-4b24-a777-214ed967695f +--------------------------------------+----------------------------------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------------------------------+ | INTERNAL-NETWORK network | 192.168.120.14 | | OS-DCF:diskConfig | AUTO | | OS-EXT-AZ:availability_zone | XXXXXXXXXXX | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2017-10-19T15:30:57.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2017-10-19T15:30:47Z | | description | zabbix-intercon | | flavor:disk | 40 | | flavor:ephemeral | 0 | | flavor:extra_specs | {} | | flavor:original_name | m1.medium | | flavor:ram | 4096 | | flavor:swap | 0 | | flavor:vcpus | 2 | | hostId | 723f077d53d4182fcb76bb87c7fc16efd7ad5fea5d939a9d1a61cb07 | | id | 5249185f-0a8e-4b24-a777-214ed967695f | | image | Centos-7 (d3f18d34-bb00-44e1-85cd-71bfae7ca24a) | | key_name | saulo-fcm360_id_rsa | | locked | False | | metadata | {} | | name | zabbix-intercon | | os-extended-volumes:volumes_attached | [{"id": "4a30e8a4-86c3-4cef-9f89-5bccefd9df22", "delete_on_termination": false}] | | progress | 0 | | security_groups | Zabbix Access, default | | status | ACTIVE | | tags | [] | | tenant_id | e082484fc4004da78407a0c0857ae8a9 | | updated | 2018-01-31T00:06:36Z | | user_id | ff9a3af4473c480cacd0c4e18d68edb4 | +--------------------------------------+----------------------------------------------------------------------------------+ Ant help ? Saulo Augusto Silva sauloaugustosilva at gmail.com To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1746509/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
