Ha, yeah have come across this too in our tempest runs. Haven’t had a chance to look into it yet. But I think you are on the right track. Just need to modify the tests.
Cheers, Sam > On 20 May 2016, at 1:25 AM, Mike Dorman <mdor...@godaddy.com> wrote: > > We are struggling a bit with the "Send up AZ and instance_name in system > metadata” patch. It works fine under normal circumstances, but if there is a > scheduler error (No valid host was found), then there is a failure when > transitioning the instance to an error state, and the instance gets stuck in > scheduling forever: > > 2016-05-19 08:15:07.677 24589 WARNING nova.scheduler.utils > [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA > f48e57277a7a484290ba9afdc49a21a9 - - -] [instance: > 71009301-7377-49f7-9f9e-f4e718bd5c83] Setting instance to ERROR state. > 2016-05-19 08:15:07.679 24589 DEBUG nova.objects.instance > [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA > f48e57277a7a484290ba9afdc49a21a9 - - -] Lazy-loading `system_metadata' on > Instance uuid 71009301-7377-49f7-9f9e-f4e718bd5c83 obj_load_attr > /usr/lib/python2.7/site-packages/nova/objects/instance.py:873 > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA > f48e57277a7a484290ba9afdc49a21a9 - - -] Exception during message handling: > Object action obj_load_attr failed because: attribute id not lazy-loadable > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher Traceback > (most recent call last): > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line > 142, in _dispatch_and_reply > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > executor_callback)) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line > 186, in _dispatch > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > executor_callback) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line > 129, in _do_dispatch > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher result > = func(ctxt, **new_args) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 752, in > build_instances > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher exc, > request_spec) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 625, in > _set_vm_state_and_notify > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ex, > request_spec, self.db) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 111, in > set_vm_state_and_notify > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > instance.save() > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, > in wrapper > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher return > fn(self, *args, **kwargs) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 615, in save > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > self._ensure_cells_system_metadata() > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 555, in > _ensure_cells_system_metadata > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > self.system_metadata[u'instance_name'] = self.name > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 224, in name > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > base_name = CONF.instance_name_template % self.id > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in > getter > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > self.obj_load_attr(name) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 864, in > obj_load_attr > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > reason='attribute %s not lazy-loadable' % attrname) > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > ObjectActionError: Object action obj_load_attr failed because: attribute id > not lazy-loadable > 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher > > > The problem is that _ensure_cells_system_metadata is referencing self.name > before the object is actually loaded from the database, so only uuid and I > think vm_state are set at that point. The name accessor method references > self.id, which (for some reason) is not lazy loadable. > > So I tried moving the _ensure_cells_system_metadata later in the save method, > after the object is loaded from the database (here: > https://github.com/openstack/nova/blob/stable/liberty/nova/objects/instance.py#L676 > ) That seems to work in practice, but it causes some of the tox tests to > fail: https://gist.github.com/misterdorm/cc7dfd235ebcc2a23009b9115b58e4d5 > > Anyways, I’m at a bit of a loss here and curious if anybody might have some > better insights. > > Thanks, > Mike > > > > From: Sam Morrison <sorri...@gmail.com> > Date: Wednesday, May 4, 2016 at 6:23 PM > To: Mike Dorman <mdor...@godaddy.com> > Cc: OpenStack Operators <openstack-operators@lists.openstack.org> > Subject: Re: [Openstack-operators] Nova cells patches against Liberty > > > Hi Mike, > > I’ve also been working on these and have some updated patches at: > > https://github.com/NeCTAR-RC/nova/commits/stable/liberty-cellsv1 > > There are a couple of patches that you have in your tree that need updating > for Liberty. Mainly around supporting the v2.1 API and more things moved to > objects. I have also written some tests for some more of them too. I haven’t > tested all of these functionally > yet but they pass all tox tests. > > Cheers, > Sam > > > > > On 5 May 2016, at 4:19 AM, Mike Dorman <mdor...@godaddy.com> wrote: > > I went ahead and pulled out the Nova cells patches we’re running against > Liberty so that others can use them if so desired. > > https://github.com/godaddy/openstack-nova-patches > > Usual disclaimers apply here, your mileage may vary, these may not work as > expected in your environment, etc. We have tested these at a basic level > (unit tests), but are not running these for Liberty in real production yet. > > Mike > > > > > > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > > > > _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
