I’m working on upgrading our nova to Juno. First I upgrade the control infrastructure, conductor,api,scheduler etc. and migrate the DB I also set the following on the control hosts [upgrade_levels] compute=icehouse
I don’t touch the nodes running nova-compute. Things seem to work however when I start a nova-compute I get the following when it registers its service: Line in question: nova/virt/libvirt/driver.py def _set_host_enabled() service = service_obj.Service.get_by_compute_host(ctx, CONF.host) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported']) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported']) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version) …. REPEATED…. 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=self.retry) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver timeout=timeout, retry=retry) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 436, in send 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 422, in _send 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 746, in topic_send 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver exchange_name=exchange_name, retry=retry) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 717, in publisher_send 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.ensure(_error_callback, _publish, retry=retry) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 617, in ensure 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver ret, channel = autoretry_method() 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, **kwargs) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__ 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, channel=channels[0], **kwargs), channels[0] 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 714, in _publish 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver publisher = cls(self.conf, channel, topic=topic, **kwargs) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 381, in __init__ 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver **options) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 329, in __init__ 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.reconnect(channel) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 337, in reconnect 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver routing_key=self.routing_key) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__ 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.revive(self._channel) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.declare() 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.exchange.declare() 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver nowait=nowait, passive=passive, 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver (40, 11), # Channel.exchange_declare_ok 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.channel_id, allowed_methods) 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line 240, in _wait_method 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.method_reader.read_method() 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver raise m 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError: maximum recursion depth exceeded while calling a Python object In conductor I see the following (after adding a debug log message): 2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439 (repeats a bunch more times) What I can’t figure out is why it’s asking for a target version of 1.3, icehouse service object is 1.2 and juno service object is 1.4 Anyone come across this? Cheers, Sam _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
