Hi All, We are experiencing a problem with our OpenStack Folsom installation on Ubuntu 12.04, using nova-network: instances are NOT created, and deleting instances in error state fails.
First of all: the installation was in "production" for almost one
year, and we didn't change anything. The only change made was updating
some packages from ubuntu and from ubuntu-cloud archives.
(PS: The following text is formatted as restructured text)
In details what happen is:
* The instance is created `Thu Oct 31 18:01:00 CET 2013`::
2013-10-31 18:01:02 INFO nova.api.openstack.wsgi
[req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4
0b41ff4c9eec47ac9ba7ee2680a693ce] GET
http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d
2013-10-31 18:01:02 INFO nova.api.openstack.wsgi
[req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4
0b41ff4c9eec47ac9ba7ee2680a693ce]
http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d
returned with HTTP 200
2013-10-31 18:01:02 INFO nova.osapi_compute.wsgi.server
[req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4
0b41ff4c9eec47ac9ba7ee2680a693ce] 130.60.24.9 - - [31/Oct/2013 18:01:02] "GET
/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d
HTTP/1.1" 200 1531 0.225107
* On the compute node, I see multiple times the following trace::
2013-10-31 18:01:26 931 INFO nova.compute.resource_tracker [-]
Compute_service record updated for node-09-01-00
2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free ram
(MB): 17389
2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free disk
(GB): 136
2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free
VCPUS: 0
2013-10-31 18:02:34 931 INFO nova.compute.resource_tracker [-]
Compute_service record updated for node-09-01-00
2013-10-31 18:02:41 931 ERROR nova.openstack.common.rpc.amqp [-]
Exception during message handling
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp Traceback
(most recent call last):
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276,
in _process_data
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp rval =
self.proxy.dispatch(ctxt, version, method, **args)
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 145, in dispatch
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return
getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp temp_level,
payload)
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return
f(*args, **kw)
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 390, in
refresh_instance_security_rules
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return
self.driver.refresh_instance_security_rules(instance)
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2243, in
refresh_instance_security_rules
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
self.firewall_driver.refresh_instance_security_rules(instance)
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 446, in
refresh_instance_security_rules
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
self.do_refresh_instance_rules(instance)
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 463, in
do_refresh_instance_rules
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp network_info
= self.network_infos[instance['id']]
2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp KeyError:
38331
This backtrace does not seem to be relevant, as I've already seen
it in log of instances that started without problems.
* After a while, IPs are assigned to the instance. I don't see
anything in the ``nova-network.log``, but I can see the IPs in the
``nova list`` output. I also see the following errors in the compute
node::
2013-10-31 18:04:04 931 ERROR nova.openstack.common.rpc.common [-] Timed
out waiting for RPC response: timed out
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common Traceback
(most recent call last):
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 518, in ensure
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return
method(*args, **kwargs)
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 596, in _consume
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in
drain_events
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in
drain_events
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in
drain_events
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return
self.wait_multi(self.channels.values(), timeout=timeout)
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in
wait_multi
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
chanmap.keys(), allowed_methods, timeout=timeout)
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in
_wait_multiple
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common channel,
method_sig, args, content = read_timeout(timeout)
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in
read_timeout
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common return
self.method_reader.read_method()
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line
221, in read_method
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common raise m
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common timeout:
timed out
2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
2013-10-31 18:04:04 ERROR nova.compute.manager
[req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4
0b41ff4c9eec47ac9ba7ee2680a693ce] [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] Instance failed network setup
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] Traceback (most recent call last):
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in
_allocate_network
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] requested_networks=requested_networks)
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] res = f(self, context, *args,
**kwargs)
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in
allocate_for_instance
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] 'args': args})
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line
108, in call
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] return _get_impl().call(cfg.CONF,
context, topic, msg, timeout)
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 718, in call
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] rpc_amqp.get_connection_pool(conf,
Connection))
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 369,
in call
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] rv = list(rv)
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 330,
in __iter__
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] self.done()
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] self.gen.next()
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 327,
in __iter__
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] self._iterator.next()
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 601, in iterconsume
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] yield self.ensure(_error_callback,
_consume)
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 521, in ensure
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] error_callback(e)
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 582, in _error_callback
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] raise rpc_common.Timeout()
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] Timeout: Timeout while waiting on RPC
response.
2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
ece6db13-10ee-4808-9646-57ab285ca33d]
2013-10-31 18:04:04 INFO nova.compute.resource_tracker
[req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4
0b41ff4c9eec47ac9ba7ee2680a693ce] Aborting claim: [Claim
ece6db13-10ee-4808-9646-57ab285ca33d: 14336 MB memory, 100 GB disk, 8 VCPUS]
and then again::
2013-10-31 18:07:04 ERROR nova.compute.manager
[req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4
0b41ff4c9eec47ac9ba7ee2680a693ce] [instance:
ece6db13-10ee-4808-9646-57ab285ca33d] Build error: ['Traceback (most recent
call last):\n', ' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 498, in
_run_instance\n requested_networks)\n', ' File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in
_allocate_network\n requested_networks=requested_networks)\n', ' File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper\n
res = f(self, context, *args, **kwargs)\n', ' File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in
allocate_for_instance\n \'args\': args})\n', ' File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line
108, in call\n return _get_impl().call(cfg.CONF, context, topic, msg,
timeout)\n', ' File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 718, in call\n rpc_amqp.get_connection_pool(conf, Connection))\n', '
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
369, in call\n rv = list(rv)\n', ' File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 330,
in __iter__\n self.done()\n', ' File "/usr/lib/python2.7/contextlib.py",
line 24, in __exit__\n self.gen.next()\n', ' File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 327,
in __iter__\n self._iterator.next()\n', ' File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 601, in iterconsume\n yield self.ensure(_error_callback, _consume)\n',
' File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 521, in ensure\n error_callback(e)\n', ' File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 582, in _error_callback\n raise rpc_common.Timeout()\n', 'Timeout:
Timeout while waiting on RPC response.\n']
2013-10-31 18:07:05 931 ERROR nova.openstack.common.rpc.amqp [-]
Exception during message handling
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp Traceback
(most recent call last):
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276,
in _process_data
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rval =
self.proxy.dispatch(ctxt, version, method, **args)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 145, in dispatch
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp temp_level,
payload)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
f(*args, **kw)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in
decorated_function
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp pass
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in
decorated_function
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in
decorated_function
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
kwargs['instance']['uuid'], e, sys.exc_info())
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in
decorated_function
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in
run_instance
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
do_run_instance()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp retval =
f(*args, **kwargs)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in
do_run_instance
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
admin_password, is_first_time, instance)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in
_run_instance
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self._set_instance_error_state(context, instance['uuid'])
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in
_run_instance
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
is_first_time, request_spec, filter_properties)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 547, in
_reschedule_or_reraise
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self._deallocate_network(context, instance)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 783, in
_deallocate_network
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self.network_api.deallocate_for_instance(context, instance)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 261, in
deallocate_for_instance
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp 'args':
args})
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line
108, in call
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
_get_impl().call(cfg.CONF, context, topic, msg, timeout)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 718, in call
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
rpc_amqp.get_connection_pool(conf, Connection))
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 369,
in call
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rv = list(rv)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 330,
in __iter__
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.done()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 327,
in __iter__
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
self._iterator.next()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 601, in iterconsume
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp yield
self.ensure(_error_callback, _consume)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 521, in ensure
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
error_callback(e)
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 582, in _error_callback
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp raise
rpc_common.Timeout()
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp Timeout:
Timeout while waiting on RPC response.
2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
* After this, the instance is in ERROR state. If I delete it, it
usually only release the IPs, and only after a while, if I re-run
``nova delete``, it *sometimes* actually delete the instance.
This is driving me crazy, I don't really know where to look for
hints. I guess the problem is in the network node, but I can't
understand anything from the logs.
can someone help me with this?
Thank you in advance
.a.
--
antonio.s.mess...@gmail.com
antonio.mess...@uzh.ch +41 (0)44 635 42 22
GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland
signature.asc
Description: PGP signature
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
