Hello KK, I see that your config files are up dated, for example lock_path = /var/lib/nova/tmp and auth_password to auth_type. Are the endpoint versions correct? Just taking a wild guess.
Best regards, Venu [image: ThoughtWorks] <http://www.thoughtworks.com/> --- *“Excellence is never an accident. It is always the result of high intention, sincere effort, and intelligent execution.... ― Aristotle* On Wed, Apr 20, 2016 at 2:57 PM, K K <[email protected]> wrote: > Hello. > > I upgraded release OpenStack from Kilo to Mitaka. > > But now I can't start any Instance. At the compute node nova daemons said: > > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > [req-86e4ab2e-229a-438d-87d2-d9ce9f749dbc 79401c3a746740ef826762ca9eaeb207 > 3029462986ac400ca2c2c6ac4c9af5a7 - - -] Exception during message handling: > Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9 > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > Traceback (most recent call last): > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line > 138, in _dispatch_and_reply > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > incoming.message)) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line > 185, in _dispatch > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return > self._do_dispatch(endpoint, method, ctxt, args) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line > 127, in _do_dispatch > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result > = func(ctxt, **new_args) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher payload) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > self.force_reraise() > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > six.reraise(self.type_, self.value, self.tb) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return > f(self, context, *args, **kw) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 359, in > decorated_function > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > LOG.warning(msg, e, instance=instance) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > self.force_reraise() > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > six.reraise(self.type_, self.value, self.tb) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 328, in > decorated_function > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return > function(self, context, *args, **kwargs) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in > decorated_function > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return > function(self, context, *args, **kwargs) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in > decorated_function > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > kwargs['instance'], e, sys.exc_info()) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > self.force_reraise() > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > six.reraise(self.type_, self.value, self.tb) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in > decorated_function > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return > function(self, context, *args, **kwargs) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2598, in > start_instance > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > self._power_on(context, instance) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2566, in > _power_on > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > network_info = self.network_api.get_instance_nw_info(context, instance) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return > func(self, context, *args, **kwargs) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 376, in > get_instance_nw_info > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > **kwargs) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 253, in > get_instance_nw_info > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result > = self._get_instance_nw_info(context, instance, **kwargs) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 385, in > _get_instance_nw_info > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher nw_info > = self.network_rpcapi.get_instance_nw_info(context, **args) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 209, in > get_instance_nw_info > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > host=host, project_id=project_id) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, > in call > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > retry=self.retry) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in > _send > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > timeout=timeout, retry=retry) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 470, in send > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > retry=retry) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 459, in _send > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result > = self._waiter.wait(msg_id, timeout) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 342, in wait > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher message > = self.waiters.get(msg_id, timeout=timeout) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 244, in get > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher 'to > message ID %s' % msg_id) > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > MessagingTimeout: Timed out waiting for a reply to message ID > cb754491dfb14d8793ca6ee05ae420c9 > 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher > 2016-04-20 12:41:15.432 226509 DEBUG oslo_concurrency.lockutils > [req-793ec582-610d-4702-a975-c642bd7fe799 - - - - -] Releasing semaphore > "refresh_cache-866156cf-f86b-43c7-9253-68dc1f939c50" lock > /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228 > > All services has started up (even rabbitmq at controller node). > Communications are success between nodes. > > At the controller node I have messages: > > ==> rabbitmq/[email protected] <== > > =INFO REPORT==== 20-Apr-2016::12:40:16 === > accepting AMQP connection <0.3201.0> (10.30.16.129:51314 -> > 10.30.17.10:5672) > > ==> nova/nova-scheduler.log <== > 2016-04-20 12:40:16.499 4099 INFO nova.scheduler.host_manager > [req-a010da22-3ea1-4019-82ce-5e434ca74692 - - - - -] Successfully synced > instances from host 'compute01'. > > That's config for nova at compute node: > > [DEFAULT] > dhcpbridge_flagfile=/etc/nova/nova.conf > dhcpbridge=/usr/bin/nova-dhcpbridge > log-dir=/var/log/nova > state_path=/var/lib/nova > lock_path=/var/lock/nova > force_dhcp_release=True > libvirt_use_virtio_for_bridges=True > debug=true > api_paste_config=/etc/nova/api-paste.ini > enabled_apis=osapi_compute,metadata > rpc_backend = rabbit > auth_strategy = keystone > my_ip = 10.30.16.129 > linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver > firewall_driver = nova.virt.firewall.NoopFirewallDriver > instance_usage_audit = True > instance_usage_audit_period = hour > notify_on_state_change = vm_and_task_state > notification_driver = messagingv2 > block_device_allocate_retries=28800 > block_device_allocate_retries_interval=3 > allow_resize_to_same_host = True > [vnc] > enabled = True > keymap = en-us > vncserver_listen = 0.0.0.0 > vncserver_proxyclient_address = 10.30.16.129 > novncproxy_base_url = http://controller:6080/vnc_auto.html > [oslo_messaging_rabbit] > rabbit_host = controller > rabbit_userid = openstack > rabbit_password = PASSWORD > [keystone_authtoken] > auth_uri = http://controller:5000/v2.0 > identity_uri = http://controller:35357 > admin_tenant_name = service > admin_user = nova > admin_password = PASSWORD > > auth_url = http://controller:35357 > [glance] > [oslo_concurrency] > lock_path = /var/lib/nova/tmp > [neutron] > url = http://controller:9696 > auth_strategy = keystone > admin_auth_url = http://controller:35357/v2.0 > admin_tenant_name = service > admin_username = neutron > admin_password = PASSWORD > [cache] > Enabled = true > debug_cache_backend = False > [libvirt] > hw_disk_discard = unmap > libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf > > live_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED,VIR_MIGRATE_NON_SHARED_INC > block_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE, > VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_NON_SHARED_INC > disk_cachemodes="network=writeback" > > Neutron's config @compute node: > > [DEFAULT] > rpc_backend = rabbit > auth_strategy = keystone > core_plugin = ml2 > service_plugins = > notify_nova_on_port_status_changes = True > notify_nova_on_port_data_changes = True > nova_url = http://controller:8774/v2 > verbose = True > debug = false > [matchmaker_redis] > [matchmaker_ring] > [quotas] > [agent] > root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf > [keystone_authtoken] > auth_uri = http://controller:5000/v2.0 > identity_uri = http://controller:35357 > admin_tenant_name = service > admin_user = neutron > admin_password = PASSWORD > > [database] > connection = mysql://neutron:PASSWORD@controller/neutron > [nova] > auth_url = http://controller:35357 > auth_type = password > project_domain_id = default > user_domain_id = default > region_name = RegionOne > project_name = service > user-name = nova > password = PASSWORD > [oslo_concurrency] > lock_path = $state_path/lock > [oslo_policy] > [oslo_messaging_amqp] > [oslo_messaging_qpid] > [oslo_messaging_rabbit] > rabbit_host = controller > rabbit_userid = openstack > rabbit_password = PASSWORD > > nova's config @controller node: > > [DEFAULT] > dhcpbridge_flagfile=/etc/nova/nova.conf > dhcpbridge=/usr/bin/nova-dhcpbridge > log-dir=/var/log/nova > state_path=/var/lib/nova > lock_path=/var/lock/nova > force_dhcp_release=True > libvirt_use_virtio_for_bridges=True > debug=false > api_paste_config=/etc/nova/api-paste.ini > enabled_apis=osapi_compute,metadata > rpc_backend = rabbit > auth_strategy = keystone > my_ip = 10.30.17.10 > linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver > firewall_driver = nova.virt.firewall.NoopFirewallDriver > block_device_allocate_retries=28800 > block_device_allocate_retries_interval=3 > allow_resize_to_same_host = True > [api_database] > connection = mysql+pymysql://nova:PASSWORD@controller > /nova_api?charset=utf8 > [database] > connection = mysql://nova:PASSWORD@controller/nova > [oslo_messaging_rabbit] > rabbit_host = controller > rabbit_userid = openstack > rabbit_password = PASSWORD > [keystone_authtoken] > auth_uri = http://controller:5000/v2.0 > identity_uri = http://controller:35357 > admin_tenant_name = service > admin_user = nova > admin_password = PASSWORD > > [glance] > [oslo_concurrency] > lock_path = /var/lib/nova/tmp > [neutron] > url = http://controller:9696 > auth_strategy = keystone > admin_auth_url = http://controller:35357/v2.0 > admin_tenant_name = service > admin_username = neutron > admin_password = PASSWORD > service_metadata_proxy = True > metadata_proxy_shared_secret = SHARED_SECRET > [libvirt] > hw_disk_discard = unmap > > live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED > > Neutron's password @controller node: > > [DEFAULT] > rpc_backend = rabbit > auth_strategy = keystone > core_plugin = ml2 > service_plugins = router > notify_nova_on_port_status_changes = True > notify_nova_on_port_data_changes = True > nova_url = http://controller:8774/v2 > verbose = True > [matchmaker_redis] > [matchmaker_ring] > [quotas] > [agent] > root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf > [keystone_authtoken] > auth_uri = http://controller:5000/v2.0 > identity_uri = http://controller:35357 > admin_tenant_name = service > admin_user = neutron > admin_password = PASSWORD > [database] > connection = mysql://neutron:PASSWORD@controller/neutron > [nova] > auth_uri = http://controller:5000/v2.0 > identity_uri = http://controller:35357 > admin_tenant_name = service > admin_user = nova > admin_password = PASSWORD > [oslo_concurrency] > lock_path = $state_path/lock > [oslo_policy] > [oslo_messaging_amqp] > [oslo_messaging_qpid] > [oslo_messaging_rabbit] > rabbit_host = controller > rabbit_userid = openstack > rabbit_password = PASSWORD > > Do you have any idea? Google didn't help ( > > > > -- > Konstantin > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
