Public bug reported: Description =========== Devstack Queens cirros instance launch failed with the error "MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a."
Creation of Network, router.. all went well. Launch instance task failed. Steps to reproduce ================== Install Devstack from stable/queens branch and launch instance Environment =========== Ubuntu Server 16.04.4 LTS Devstack stable/queens stack@stack:/opt/devstack$ git log -1 commit bb0c101c0c7352b0fadf0e13d4c47f59437db454 Merge: 661b186 abe9fd8 Author: Zuul <[email protected]> Date: Tue Mar 13 02:28:44 2018 +0000 Merge "Cap max microversions for queens" into stable/queens Logs & Configs ============== nova-compute.log ----------------- Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR os_vif [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m#033[01;31mFailed to plug vif VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b')#033[00m: ProcessExecutionError: Unexpected error while running command. Mar 19 06:29:32 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0 Mar 19 06:29:32 stack nova-compute[6834]: Exit code: 1 Mar 19 06:29:32 stack nova-compute[6834]: Stdout: u'' Mar 19 06:29:32 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mTraceback (most recent call last): Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/os_vif/__init__.py", line 77, in plug Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m plugin.plug(vif, instance_info) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 209, in plug Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m self._plug_bridge(vif, instance_info) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 161, in _plug_bridge Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m linux_net.ensure_bridge(vif.bridge_name) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 207, in _wrap Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m return self.channel.remote_call(name, args, kwargs) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 202, in remote_call Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m raise exc_type(*result[2]) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mProcessExecutionError: Unexpected error while running command. Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mCommand: brctl setageing qbr41533537-1b 0 Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mExit code: 1 Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStdout: u'' Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR nova.virt.libvirt.driver [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to start libvirt guest#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command. Mar 19 06:29:33 stack nova-compute[6834]: #033[01;31mERROR nova.compute.manager [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mInstance failed to spawn#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command. Mar 19 06:29:33 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0 Mar 19 06:29:33 stack nova-compute[6834]: Exit code: 1 Mar 19 06:29:33 stack nova-compute[6834]: Stdout: u'' Mar 19 06:29:33 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last): Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/compute/manager.py", line 2238, in _build_resources Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m yield resources Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/compute/manager.py", line 2022, in _build_and_run_instance Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m block_device_info=block_device_info) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3047, in spawn Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m destroy_disks_on_failure=True) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5536, in _create_domain_and_network Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m destroy_disks_on_failure) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self.force_reraise() Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m six.reraise(self.type_, self.value, self.tb) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5495, in _create_domain_and_network Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self.plug_vifs(instance, network_info) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 822, in plug_vifs Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self.vif_driver.plug(instance, vif) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 769, in plug Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self._plug_os_vif(instance, vif_obj) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 749, in _plug_os_vif Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m raise exception.InternalError(msg) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mInternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command. Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mCommand: brctl setageing qbr41533537-1b 0 Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mExit code: 1 Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStdout: u'' Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m nova-conductor.log ------------------- Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mError from last host: stack (node stack): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1840, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2108, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.\nCommand: brctl setageing qbr41533537-1b 0\nExit code: 1\nStdout: u''\nStderr: u'set ageing time failed: Numerical result out of range\\n'\n"]#033[00m Mar 19 06:29:35 stack nova-conductor[5910]: #033[00;32mDEBUG nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mRescheduling: True#033[00m #033[00;33m{{(pid=6969) build_instances /opt/stack/nova/nova/conductor/manager.py:570}}#033[00m Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m#033[01;33mFailed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a. Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;33mSetting instance to ERROR state.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a. Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to deallocate networks#033[00m: Unauthorized: Unknown auth type: None Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last): Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/conductor/manager.py", line 366, in _cleanup_allocated_networks Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m context, instance, requested_networks=requested_networks) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1270, in deallocate_for_instance Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self._unbind_ports(context, ports_to_skip, neutron) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 527, in _unbind_ports Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m port_client = get_client(context, admin=True) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 153, in get_client Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m _ADMIN_AUTH = _load_auth_plugin(CONF) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 76, in _load_auth_plugin Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m raise neutron_client_exc.Unauthorized(message=err_msg) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mUnauthorized: Unknown auth type: None Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m Mar 19 06:29:36 stack [email protected][24084]: #033[00;36mINFO nova.api.openstack.requestlog [#033[01;36mNone req-2685a1bf-9098-49f7-8f86-00a01bab5a05 #033[00;36mdemo admin#033[00;36m] #033[01;35m#033[00;36m10.39.18.67 "GET /compute/v2.1" status: 302 len: None microversion: - time: 0.004733#033[00m ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1756845 Title: Queens Devstack instance creation failure "Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a" Status in OpenStack Compute (nova): New Bug description: Description =========== Devstack Queens cirros instance launch failed with the error "MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a." Creation of Network, router.. all went well. Launch instance task failed. Steps to reproduce ================== Install Devstack from stable/queens branch and launch instance Environment =========== Ubuntu Server 16.04.4 LTS Devstack stable/queens stack@stack:/opt/devstack$ git log -1 commit bb0c101c0c7352b0fadf0e13d4c47f59437db454 Merge: 661b186 abe9fd8 Author: Zuul <[email protected]> Date: Tue Mar 13 02:28:44 2018 +0000 Merge "Cap max microversions for queens" into stable/queens Logs & Configs ============== nova-compute.log ----------------- Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR os_vif [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m#033[01;31mFailed to plug vif VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b')#033[00m: ProcessExecutionError: Unexpected error while running command. Mar 19 06:29:32 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0 Mar 19 06:29:32 stack nova-compute[6834]: Exit code: 1 Mar 19 06:29:32 stack nova-compute[6834]: Stdout: u'' Mar 19 06:29:32 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mTraceback (most recent call last): Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/os_vif/__init__.py", line 77, in plug Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m plugin.plug(vif, instance_info) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 209, in plug Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m self._plug_bridge(vif, instance_info) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 161, in _plug_bridge Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m linux_net.ensure_bridge(vif.bridge_name) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 207, in _wrap Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m return self.channel.remote_call(name, args, kwargs) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 202, in remote_call Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m raise exc_type(*result[2]) Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mProcessExecutionError: Unexpected error while running command. Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mCommand: brctl setageing qbr41533537-1b 0 Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mExit code: 1 Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStdout: u'' Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00mStderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:32 stack nova-compute[6834]: ERROR os_vif #033[01;35m#033[00m Mar 19 06:29:32 stack nova-compute[6834]: #033[01;31mERROR nova.virt.libvirt.driver [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to start libvirt guest#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command. Mar 19 06:29:33 stack nova-compute[6834]: #033[01;31mERROR nova.compute.manager [#033[01;36mNone req-3fb97b01-cbcd-4b95-ac44-87b6afa6ebab #033[00;36mservice nova#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mInstance failed to spawn#033[00m: InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command. Mar 19 06:29:33 stack nova-compute[6834]: Command: brctl setageing qbr41533537-1b 0 Mar 19 06:29:33 stack nova-compute[6834]: Exit code: 1 Mar 19 06:29:33 stack nova-compute[6834]: Stdout: u'' Mar 19 06:29:33 stack nova-compute[6834]: Stderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last): Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/compute/manager.py", line 2238, in _build_resources Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m yield resources Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/compute/manager.py", line 2022, in _build_and_run_instance Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m block_device_info=block_device_info) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3047, in spawn Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m destroy_disks_on_failure=True) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5536, in _create_domain_and_network Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m destroy_disks_on_failure) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self.force_reraise() Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m six.reraise(self.type_, self.value, self.tb) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5495, in _create_domain_and_network Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self.plug_vifs(instance, network_info) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 822, in plug_vifs Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self.vif_driver.plug(instance, vif) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 769, in plug Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self._plug_os_vif(instance, vif_obj) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 749, in _plug_os_vif Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m raise exception.InternalError(msg) Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mInternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command. Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mCommand: brctl setageing qbr41533537-1b 0 Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mExit code: 1 Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStdout: u'' Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mStderr: u'set ageing time failed: Numerical result out of range\n' Mar 19 06:29:33 stack nova-compute[6834]: ERROR nova.compute.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m nova-conductor.log ------------------- Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mError from last host: stack (node stack): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1840, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2108, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:0a:69:57,bridge_name='qbr41533537-1b',has_traffic_filtering=True,id=41533537-1b6c-4b64-8eff-2c05d281a564,network=Network(9148e3c5-5a24-4ef5-a569-ec7f1301759b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap41533537-1b'). Got error: Unexpected error while running command.\nCommand: brctl setageing qbr41533537-1b 0\nExit code: 1\nStdout: u''\nStderr: u'set ageing time failed: Numerical result out of range\\n'\n"]#033[00m Mar 19 06:29:35 stack nova-conductor[5910]: #033[00;32mDEBUG nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[00;32m] #033[01;35m#033[00;32mRescheduling: True#033[00m #033[00;33m{{(pid=6969) build_instances /opt/stack/nova/nova/conductor/manager.py:570}}#033[00m Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m#033[01;33mFailed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a. Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;33mWARNING nova.scheduler.utils [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;33m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;33mSetting instance to ERROR state.#033[00m: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 833e1bf6-7ea8-413f-94b7-0c87528a8d0a. Mar 19 06:29:35 stack nova-conductor[5910]: #033[01;31mERROR nova.conductor.manager [#033[01;36mNone req-f0dbe20e-637f-4ac8-b6c5-8f7c6ac3da11 #033[00;36mdemo admin#033[01;31m] #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[01;31mFailed to deallocate networks#033[00m: Unauthorized: Unknown auth type: None Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mTraceback (most recent call last): Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/conductor/manager.py", line 366, in _cleanup_allocated_networks Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m context, instance, requested_networks=requested_networks) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1270, in deallocate_for_instance Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m self._unbind_ports(context, ports_to_skip, neutron) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 527, in _unbind_ports Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m port_client = get_client(context, admin=True) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 153, in get_client Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m _ADMIN_AUTH = _load_auth_plugin(CONF) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m File "/opt/stack/nova/nova/network/neutronv2/api.py", line 76, in _load_auth_plugin Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m raise neutron_client_exc.Unauthorized(message=err_msg) Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00mUnauthorized: Unknown auth type: None Mar 19 06:29:35 stack nova-conductor[5910]: ERROR nova.conductor.manager #033[01;35m[instance: 833e1bf6-7ea8-413f-94b7-0c87528a8d0a] #033[00m Mar 19 06:29:36 stack [email protected][24084]: #033[00;36mINFO nova.api.openstack.requestlog [#033[01;36mNone req-2685a1bf-9098-49f7-8f86-00a01bab5a05 #033[00;36mdemo admin#033[00;36m] #033[01;35m#033[00;36m10.39.18.67 "GET /compute/v2.1" status: 302 len: None microversion: - time: 0.004733#033[00m To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1756845/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
