Hello,
We have a working two host Folsom (2012.2.3 on Ubuntu 12.04) OpenStack
cluster with a controller and compute node. This afternoon we added a
second compute node, but attempts to spawn instances on the new node fail
with a glance client exception (see example log below).
I've triple-checked our new nova.conf and api-paste.ini files, which are
identical to what we have on the working compute node, except for the local
IP address change to metadata_host, vncserver_proxyclient_address, and
my_ip local.
When I run "nova-manage service list" the nova-compute and nova-network
services on all three machines report as status :-).
The ONLY significant difference I'm seeing is that the new compute node is
running the 2012.2.4 version of nova-network, nova-compute,
and nova-api-metadata. I wouldn't expect this to be a problem, but maybe a
bug or incompatibility was introduced in 2012.2.4? I'd like to try rolling
back to 2012.2.3, but I can't find a copy of the older .deb packages.
Am I missing something obvious? Can anyone offer a suggestion?
best,
Mark
2013-06-28 02:12:47 5290 ERROR nova.openstack.common.rpc.amqp [-] Exception
during message handling
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp Traceback
(most recent call last):
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
276, in _process_data
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp rval =
self.proxy.dispatch(ctxt, version, method, **args)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 145, in dispatch
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return
getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return
f(*args, **kw)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in
decorated_function
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp pass
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in
decorated_function
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in
decorated_function
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
kwargs['instance']['uuid'], e, sys.exc_info())
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in
decorated_function
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in
run_instance
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
do_run_instance()
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp retval =
f(*args, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in
do_run_instance
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
admin_password, is_first_time, instance)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in
_run_instance
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
self._set_instance_error_state(context, instance['uuid'])
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in
_run_instance
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
is_first_time, request_spec, filter_properties)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 503, in
_run_instance
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
injected_files, admin_password)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 756, in
_spawn
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
block_device_info)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return
f(*args, **kw)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1099,
in spawn
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
admin_pass=admin_password)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1365,
in _create_image
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
project_id=instance['project_id'])
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line
131, in cache
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp *args,
**kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line
178, in create_image
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
prepare_template(target=base, *args, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 795, in inner
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp retval =
f(*args, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line
122, in call_if_not_exists
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
fetch_func(target=target, *args, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in
fetch_image
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
images.fetch_to_raw(context, image_id, target, user_id, project_id)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in
fetch_to_raw
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
fetch(context, image_href, path_tmp, user_id, project_id)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 82, in fetch
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
image_service.download(context, image_id, image_file)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 222, in
download
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
_reraise_translated_image_exception(image_id)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 220, in
download
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
image_chunks = self._client.call(context, 1, 'data', image_id)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 138, in call
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return
getattr(client.images, method)(*args, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 108, in
data
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp resp,
body = self.api.raw_request('GET', '/v1/images/%s' % image_id)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 191,
in raw_request
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return
self._http_request(url, method, **kwargs)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 158,
in _http_request
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp raise
exc.from_response(resp)
2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
