We are struggling a bit with the "Send up AZ and instance_name in system metadata” patch. It works fine under normal circumstances, but if there is a scheduler error (No valid host was found), then there is a failure when transitioning the instance to an error state, and the instance gets stuck in scheduling forever:
2016-05-19 08:15:07.677 24589 WARNING nova.scheduler.utils [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] [instance: 71009301-7377-49f7-9f9e-f4e718bd5c83] Setting instance to ERROR state. 2016-05-19 08:15:07.679 24589 DEBUG nova.objects.instance [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Lazy-loading `system_metadata' on Instance uuid 71009301-7377-49f7-9f9e-f4e718bd5c83 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:873 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Exception during message handling: Object action obj_load_attr failed because: attribute id not lazy-loadable 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher executor_callback)) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher executor_callback) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 752, in build_instances 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher exc, request_spec) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 625, in _set_vm_state_and_notify 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ex, request_spec, self.db) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 111, in set_vm_state_and_notify 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher instance.save() 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher return fn(self, *args, **kwargs) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 615, in save 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher self._ensure_cells_system_metadata() 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 555, in _ensure_cells_system_metadata 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher self.system_metadata[u'instance_name'] = self.name 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 224, in name 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher base_name = CONF.instance_name_template % self.id 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher self.obj_load_attr(name) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 864, in obj_load_attr 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher reason='attribute %s not lazy-loadable' % attrname) 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ObjectActionError: Object action obj_load_attr failed because: attribute id not lazy-loadable 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher The problem is that _ensure_cells_system_metadata is referencing self.name before the object is actually loaded from the database, so only uuid and I think vm_state are set at that point. The name accessor method references self.id, which (for some reason) is not lazy loadable. So I tried moving the _ensure_cells_system_metadata later in the save method, after the object is loaded from the database (here: https://github.com/openstack/nova/blob/stable/liberty/nova/objects/instance.py#L676 ) That seems to work in practice, but it causes some of the tox tests to fail: https://gist.github.com/misterdorm/cc7dfd235ebcc2a23009b9115b58e4d5 Anyways, I’m at a bit of a loss here and curious if anybody might have some better insights. Thanks, Mike From: Sam Morrison <[email protected]> Date: Wednesday, May 4, 2016 at 6:23 PM To: Mike Dorman <[email protected]> Cc: OpenStack Operators <[email protected]> Subject: Re: [Openstack-operators] Nova cells patches against Liberty Hi Mike, I’ve also been working on these and have some updated patches at: https://github.com/NeCTAR-RC/nova/commits/stable/liberty-cellsv1 There are a couple of patches that you have in your tree that need updating for Liberty. Mainly around supporting the v2.1 API and more things moved to objects. I have also written some tests for some more of them too. I haven’t tested all of these functionally yet but they pass all tox tests. Cheers, Sam On 5 May 2016, at 4:19 AM, Mike Dorman <[email protected]> wrote: I went ahead and pulled out the Nova cells patches we’re running against Liberty so that others can use them if so desired. https://github.com/godaddy/openstack-nova-patches Usual disclaimers apply here, your mileage may vary, these may not work as expected in your environment, etc. We have tested these at a basic level (unit tests), but are not running these for Liberty in real production yet. Mike _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
