On 08/29/2013 03:50 AM, Sam Stoelinga wrote:
Hi,
We have the following setup:
Folsom version
1 controller running all apis and also nova-compute
1 compute node only running nova-compute and nova-network
Problem is that when we reboot the controller the compute node
reconnects successfully according to the nova-compute and nova-network
logs, but it won't accept any messages from rabbitmq. Seems to be that
the consumer didn't actually reconnect.
I noticed that in impl_kombu.py:Connection._connect on line 457 the
variable self.consumers is an empty list. I'm not sure if this is normal.
When launching an instance the compute node never receives the message
from rabbitmq. If we use durable queues the message will stay in the
queue until nova-compute is manually restarted. And if we don't use
durable queues the queue for the compute node doesn't get
declared/created at all.
Another side note is that if we only stop rabbitmq server for 5 minutes
and later start it again. Everything works fine. So we only have this
problem when executing the command: $ reboot on the controller.
The logs look like it's normal, but in fact it's not see nova-compute.log:
2013-08-29 15:38:49 27818 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672>
2013-08-29 15:38:49 27818 ERROR nova.openstack.common.rpc.common [-]
AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672> is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
2013-08-29 15:39:03 27818 ERROR nova.service [-] Recovered model server
connection!
2013-08-29 15:39:03 27818 DEBUG nova.service [-] heart_beat rpc send...
report_state /usr/lib/python2.7/dist-packages/nova/service.py:558
2013-08-29 15:39:13 27818 DEBUG nova.service [-] heart_beat rpc send...
report_state /usr/lib/python2.7/dist-packages/nova/service.py:558
2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672>
2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.common [-]
Connected to AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672>
Anybody any suggestions on where to look or any hot fixes that can be
applied?
It's this bug, more than likely:
https://bugs.launchpad.net/oslo/+bug/856764
It's a major issue, IMO. My colleague Kevin Bringard has a patch (see
the bug) that fixes this issue. We're currently manually patching our
OpenStack services with this fix until we can get the fix into oslo.rpc.
Note, however, that the fix depends on a newer version of kombu than is
in Ubuntu 12.04 LTS at this time...
Read the discussion on the bug for more information.
Best,
-jay
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
