Hello.
I upgraded release OpenStack from Kilo to Mitaka.
But now I can't start any Instance. At the compute node nova daemons said:
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
[req-86e4ab2e-229a-438d-87d2-d9ce9f749dbc 79401c3a746740ef826762ca9eaeb207
3029462986ac400ca2c2c6ac4c9af5a7 - - -] Exception during message handling:
Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher Traceback
(most recent call last):
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138,
in _dispatch_and_reply
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
incoming.message))
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185,
in _dispatch
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127,
in _do_dispatch
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result =
func(ctxt, **new_args)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher payload)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
self.force_reraise()
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 359, in
decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
LOG.warning(msg, e, instance=instance)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
self.force_reraise()
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 328, in
decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in
decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in
decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
kwargs['instance'], e, sys.exc_info())
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
self.force_reraise()
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in
decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2598, in
start_instance
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
self._power_on(context, instance)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2566, in
_power_on
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher network_info
= self.network_api.get_instance_nw_info(context, instance)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
func(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 376, in
get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 253, in
get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result =
self._get_instance_nw_info(context, instance, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 385, in
_get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher nw_info =
self.network_rpcapi.get_instance_nw_info(context, **args)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 209, in
get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher host=host,
project_id=project_id)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in
call
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
retry=self.retry)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in
_send
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
timeout=timeout, retry=retry)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
470, in send
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher retry=retry)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
459, in _send
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result =
self._waiter.wait(msg_id, timeout)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
342, in wait
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher message =
self.waiters.get(msg_id, timeout=timeout)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
244, in get
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher 'to message
ID %s' % msg_id)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
MessagingTimeout: Timed out waiting for a reply to message ID
cb754491dfb14d8793ca6ee05ae420c9
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
2016-04-20 12:41:15.432 226509 DEBUG oslo_concurrency.lockutils
[req-793ec582-610d-4702-a975-c642bd7fe799 - - - - -] Releasing semaphore
"refresh_cache-866156cf-f86b-43c7-9253-68dc1f939c50" lock
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
All services has started up (even rabbitmq at controller node). Communications
are success between nodes.
At the controller node I have messages:
==> rabbitmq/rab...@controller.log <==
=INFO REPORT==== 20-Apr-2016::12:40:16 ===
accepting AMQP connection <0.3201.0> (10.30.16.129:51314 -> 10.30.17.10:5672)
==> nova/nova-scheduler.log <==
2016-04-20 12:40:16.499 4099 INFO nova.scheduler.host_manager
[req-a010da22-3ea1-4019-82ce-5e434ca74692 - - - - -] Successfully synced
instances from host 'compute01'.
That's config for nova at compute node:
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
log-dir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
debug=true
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 10.30.16.129
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
instance_usage_audit = True
instance_usage_audit_period = hour
notify_on_state_change = vm_and_task_state
notification_driver = messagingv2
block_device_allocate_retries=28800
block_device_allocate_retries_interval=3
allow_resize_to_same_host = True
[vnc]
enabled = True
keymap = en-us
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = 10.30.16.129
novncproxy_base_url = http://controller:6080/vnc_auto.html
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = nova
admin_password = PASSWORD
auth_url = http://controller:35357
[glance]
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[neutron]
url = http://controller:9696
auth_strategy = keystone
admin_auth_url = http://controller:35357/v2.0
admin_tenant_name = service
admin_username = neutron
admin_password = PASSWORD
[cache]
Enabled = true
debug_cache_backend = False
[libvirt]
hw_disk_discard = unmap
libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf
live_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED,VIR_MIGRATE_NON_SHARED_INC
block_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,
VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_NON_SHARED_INC
disk_cachemodes="network=writeback"
Neutron's config @compute node:
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
core_plugin = ml2
service_plugins =
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
verbose = True
debug = false
[matchmaker_redis]
[matchmaker_ring]
[quotas]
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = neutron
admin_password = PASSWORD
[database]
connection = mysql://neutron:PASSWORD@controller/neutron
[nova]
auth_url = http://controller:35357
auth_type = password
project_domain_id = default
user_domain_id = default
region_name = RegionOne
project_name = service
user-name = nova
password = PASSWORD
[oslo_concurrency]
lock_path = $state_path/lock
[oslo_policy]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
nova's config @controller node:
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
log-dir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
debug=false
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 10.30.17.10
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
block_device_allocate_retries=28800
block_device_allocate_retries_interval=3
allow_resize_to_same_host = True
[api_database]
connection = mysql+pymysql://nova:PASSWORD@controller/nova_api?charset=utf8
[database]
connection = mysql://nova:PASSWORD@controller/nova
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = nova
admin_password = PASSWORD
[glance]
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[neutron]
url = http://controller:9696
auth_strategy = keystone
admin_auth_url = http://controller:35357/v2.0
admin_tenant_name = service
admin_username = neutron
admin_password = PASSWORD
service_metadata_proxy = True
metadata_proxy_shared_secret = SHARED_SECRET
[libvirt]
hw_disk_discard = unmap
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED
Neutron's password @controller node:
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
core_plugin = ml2
service_plugins = router
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
verbose = True
[matchmaker_redis]
[matchmaker_ring]
[quotas]
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = neutron
admin_password = PASSWORD
[database]
connection = mysql://neutron:PASSWORD@controller/neutron
[nova]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = nova
admin_password = PASSWORD
[oslo_concurrency]
lock_path = $state_path/lock
[oslo_policy]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
Do you have any idea? Google didn't help (
--
Konstantin
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
