Hi Joe, I’ve managed to track this down. When Juno conductor sends back the service object, it also has an embedded compute node object. This compute node object is too new for Icehouse and so it attempts to back port it.
However for some reason instead of asking to back port the compute node object it tries to back port the service object again at the intended compute node version. This is where the loop is. There must be a bug when trying to back port embedded objects. The fix is quite simple if you are using plain juno and icehouse as there is a bug in the service object [0], it has a check when back porting the service and downgrading the compute node but it changes it to the wrong version. There is also a bug in the fixed_ip object with the embedded network object which means deletes don’t work. [1] I have a fix for that too, looking at master the code around this has changed a lot so I don’t know if this has been fixed here. I’m wondering if my patches can be applied straight to Juno? [2],[3] Cheers, Sam [0] https://bugs.launchpad.net/nova/+bug/1408496 <https://bugs.launchpad.net/nova/+bug/1408496> [1] https://bugs.launchpad.net/nova/+bug/1408498 [2] https://github.com/NeCTAR-RC/nova/commit/5c9b340b391cdbb8e3a97fc01258e59b827cb2a9 [3] https://github.com/NeCTAR-RC/nova/commit/cc20ae961578ede313add592ba81b215e9791816 > On 8 Jan 2015, at 4:47 am, Joe Gordon <[email protected]> wrote: > > > > On Tue, Jan 6, 2015 at 5:12 PM, Sam Morrison <[email protected] > <mailto:[email protected]>> wrote: > I’m working on upgrading our nova to Juno. > > First I upgrade the control infrastructure, conductor,api,scheduler etc. and > migrate the DB > I also set the following on the control hosts > [upgrade_levels] > compute=icehouse > > I don’t touch the nodes running nova-compute. > > Things seem to work however when I start a nova-compute I get the following > when it registers its service: > > Are you restarting existing icehouse nova-compute and not juno? > > Can you share sanitized copies of your config file as not sure what the issue > is based on the stacktrace alone. > > > Line in question: > nova/virt/libvirt/driver.py > def _set_host_enabled() > service = service_obj.Service.get_by_compute_host(ctx, CONF.host) > > > 2015-01-07 12:06:51.377 26596 <tel:51.377%2026596> TRACE > nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", > line 155, in call > 2015-01-07 12:06:51.377 26596 <tel:51.377%2026596> TRACE > nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, > result) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/rpc.py", line 111, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._base.deserialize_entity(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = > self._process_object(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 545, in _process_object > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > e.kwargs['supported']) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/api.py", line 280, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._manager.object_backport(context, objinst, target_version) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > target_version=target_version) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", > line 155, in call > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self.serializer.deserialize_entity(ctxt, result) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/rpc.py", line 111, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._base.deserialize_entity(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = > self._process_object(context, entity) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/objects/base.py", line 545, in _process_object > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > e.kwargs['supported']) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/api.py", line 280, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > self._manager.object_backport(context, objinst, target_version) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > target_version=target_version) > …. REPEATED…. > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", > line 152, in call > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > retry=self.retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py", > line 90, in _send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > timeout=timeout, retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 436, in send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", > line 422, in _send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 746, in topic_send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > exchange_name=exchange_name, retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 717, in publisher_send > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.ensure(_error_callback, _publish, retry=retry) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 617, in ensure > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver ret, channel > = autoretry_method() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line > 436, in _ensured > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > fun(*args, **kwargs) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line > 508, in __call__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return > fun(*args, channel=channels[0], **kwargs), channels[0] > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 714, in _publish > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver publisher = > cls(self.conf, channel, topic=topic, **kwargs) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 381, in __init__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver **options) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 329, in __init__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.reconnect(channel) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", > line 337, in reconnect > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > routing_key=self.routing_key) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line > 85, in __init__ > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.revive(self._channel) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line > 218, in revive > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.declare() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line > 105, in declare > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.exchange.declare() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line 166, > in declare > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > nowait=nowait, passive=passive, > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line 620, > in exchange_declare > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver (40, 11), # > Channel.exchange_declare_ok > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py", > line 67, in wait > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.channel_id, allowed_methods) > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line > 240, in _wait_method > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver > self.method_reader.read_method() > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File > "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py", > line 189, in read_method > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver raise m > 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError: > maximum recursion depth exceeded while calling a Python object > > > In conductor I see the following (after adding a debug log message): > 2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.2 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > 2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.3 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > 2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.2 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > 2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager > [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport > Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), > target_version=1.3 object_backport > /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 > (repeats a bunch more times) > > What I can’t figure out is why it’s asking for a target version of 1.3, > icehouse service object is 1.2 and juno service object is 1.4 > > Anyone come across this? > > We haven't seen this in any other bug reports or in our automated testing. > > > Cheers, > Sam > > > > > > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack> > Post to : [email protected] > <mailto:[email protected]> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
