Hey Mike, I have updated our patch for this https://github.com/NeCTAR-RC/nova/commit/debd88652bd9f32f91ab68dd1a3979a7cdc9ec80
Is that what you’re doing too? Cheers, Sam > On 20 May 2016, at 9:26 AM, Sam Morrison <sorri...@gmail.com> wrote: > > Ha, yeah have come across this too in our tempest runs. Haven’t had a chance > to look into it yet. But I think you are on the right track. Just need to > modify the tests. > > Cheers, > Sam > > > >> On 20 May 2016, at 1:25 AM, Mike Dorman <mdor...@godaddy.com> wrote: >> >> We are struggling a bit with the "Send up AZ and instance_name in system >> metadata” patch. It works fine under normal circumstances, but if there is >> a scheduler error (No valid host was found), then there is a failure when >> transitioning the instance to an error state, and the instance gets stuck in >> scheduling forever: >> >> 2016-05-19 08:15:07.677 24589 WARNING nova.scheduler.utils >> [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA >> f48e57277a7a484290ba9afdc49a21a9 - - -] [instance: >> 71009301-7377-49f7-9f9e-f4e718bd5c83] Setting instance to ERROR state. >> 2016-05-19 08:15:07.679 24589 DEBUG nova.objects.instance >> [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA >> f48e57277a7a484290ba9afdc49a21a9 - - -] Lazy-loading `system_metadata' on >> Instance uuid 71009301-7377-49f7-9f9e-f4e718bd5c83 obj_load_attr >> /usr/lib/python2.7/site-packages/nova/objects/instance.py:873 >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA >> f48e57277a7a484290ba9afdc49a21a9 - - -] Exception during message handling: >> Object action obj_load_attr failed because: attribute id not lazy-loadable >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher Traceback >> (most recent call last): >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line >> 142, in _dispatch_and_reply >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> executor_callback)) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line >> 186, in _dispatch >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> executor_callback) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line >> 129, in _do_dispatch >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher result >> = func(ctxt, **new_args) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 752, in >> build_instances >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher exc, >> request_spec) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 625, in >> _set_vm_state_and_notify >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ex, >> request_spec, self.db) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 111, in >> set_vm_state_and_notify >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> instance.save() >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, >> in wrapper >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher return >> fn(self, *args, **kwargs) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 615, in >> save >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> self._ensure_cells_system_metadata() >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 555, in >> _ensure_cells_system_metadata >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> self.system_metadata[u'instance_name'] = self.name >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 224, in >> name >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> base_name = CONF.instance_name_template % self.id >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, >> in getter >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> self.obj_load_attr(name) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher File >> "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 864, in >> obj_load_attr >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> reason='attribute %s not lazy-loadable' % attrname) >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> ObjectActionError: Object action obj_load_attr failed because: attribute id >> not lazy-loadable >> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher >> >> >> The problem is that _ensure_cells_system_metadata is referencing self.name >> before the object is actually loaded from the database, so only uuid and I >> think vm_state are set at that point. The name accessor method references >> self.id, which (for some reason) is not lazy loadable. >> >> So I tried moving the _ensure_cells_system_metadata later in the save >> method, after the object is loaded from the database (here: >> https://github.com/openstack/nova/blob/stable/liberty/nova/objects/instance.py#L676 >> ) That seems to work in practice, but it causes some of the tox tests to >> fail: https://gist.github.com/misterdorm/cc7dfd235ebcc2a23009b9115b58e4d5 >> >> Anyways, I’m at a bit of a loss here and curious if anybody might have some >> better insights. >> >> Thanks, >> Mike >> >> >> >> From: Sam Morrison <sorri...@gmail.com> >> Date: Wednesday, May 4, 2016 at 6:23 PM >> To: Mike Dorman <mdor...@godaddy.com> >> Cc: OpenStack Operators <openstack-operators@lists.openstack.org> >> Subject: Re: [Openstack-operators] Nova cells patches against Liberty >> >> >> Hi Mike, >> >> I’ve also been working on these and have some updated patches at: >> >> https://github.com/NeCTAR-RC/nova/commits/stable/liberty-cellsv1 >> >> There are a couple of patches that you have in your tree that need updating >> for Liberty. Mainly around supporting the v2.1 API and more things moved to >> objects. I have also written some tests for some more of them too. I haven’t >> tested all of these functionally >> yet but they pass all tox tests. >> >> Cheers, >> Sam >> >> >> >> >> On 5 May 2016, at 4:19 AM, Mike Dorman <mdor...@godaddy.com> wrote: >> >> I went ahead and pulled out the Nova cells patches we’re running against >> Liberty so that others can use them if so desired. >> >> https://github.com/godaddy/openstack-nova-patches >> >> Usual disclaimers apply here, your mileage may vary, these may not work as >> expected in your environment, etc. We have tested these at a basic level >> (unit tests), but are not running these for Liberty in real production yet. >> >> Mike >> >> >> >> >> >> >> _______________________________________________ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> >> >> >> >> > _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
