Public bug reported: Nova Version: 2.22.0
I have share nfs storage mounting the /var/lib/nova between two compute nodes. When I tried to re-sizing the instance using nova resize command, it is failing and below is the output of log 2015-10-19 05:13:15.582 14325 ERROR oslo_messaging.rpc.dispatcher [req-5cb16661-74ec-4faf-93cd-044e597cc9de d4209dcd86b84fc584f8b3b72bee0c64 da6c9fa9be0046dda47e9bd6caf3908a - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command. Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653 Exit code: 255 Stdout: u'' Stderr: u'Host key verification failed.\r\n' 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6748, in resize_instance 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher clean_shutdown=clean_shutdown) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher payload) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 377, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 286, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher migration.instance_uuid, exc_info=True) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 269, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4012, in resize_instance 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher self.instance_events.clear_events_for_instance(instance) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher self.gen.throw(type, value, traceback) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6382, in _error_out_instance_on_exception 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher raise error.inner_exception 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command. 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Exit code: 255 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stdout: u'' 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n' 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher When I checked the code, I found in /usr/lib/python2.7/dist- packages/nova/virt/libvirt/driver.py file we checked the share storage is true or not before migrating. def _is_storage_shared_with(self, dest, inst_base): # NOTE (rmk): There are two methods of determining whether we are # on the same filesystem: the source and dest IP are the # same, or we create a file on the dest system via SSH # and check whether the source system can also see it. shared_storage = (dest == self.get_host_ip_addr()) if not shared_storage: tmp_file = uuid.uuid4().hex + '.tmp' tmp_path = os.path.join(inst_base, tmp_file) LOG.debug("Temp path is") LOG.debug(tmp_path) try: utils.execute('ssh', dest, 'touch', tmp_path) if os.path.exists(tmp_path): shared_storage = True os.unlink(tmp_path) else: utils.execute('ssh', dest, 'rm', tmp_path) except Exception: pass return shared_storage But in case of share storage between compute node “shared storage = (dest == self.get_host_ip_addr())” always return false and also ssh mkdir return false and Nova resize will fail.I think checking the share storage using " shared_storage = (dest == self.get_host_ip_addr())" is not the right way. When I always return true from _is_storage_shared_with function, the Nova resize works fine. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1507521 Title: Nova Resize is failing for shared storage between Compute node Status in OpenStack Compute (nova): New Bug description: Nova Version: 2.22.0 I have share nfs storage mounting the /var/lib/nova between two compute nodes. When I tried to re-sizing the instance using nova resize command, it is failing and below is the output of log 2015-10-19 05:13:15.582 14325 ERROR oslo_messaging.rpc.dispatcher [req-5cb16661-74ec-4faf-93cd-044e597cc9de d4209dcd86b84fc584f8b3b72bee0c64 da6c9fa9be0046dda47e9bd6caf3908a - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command. Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653 Exit code: 255 Stdout: u'' Stderr: u'Host key verification failed.\r\n' 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6748, in resize_instance 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher clean_shutdown=clean_shutdown) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher payload) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 377, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 286, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher migration.instance_uuid, exc_info=True) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 269, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4012, in resize_instance 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher self.instance_events.clear_events_for_instance(instance) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher self.gen.throw(type, value, traceback) 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6382, in _error_out_instance_on_exception 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher raise error.inner_exception 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command. 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Exit code: 255 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stdout: u'' 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n' 2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher When I checked the code, I found in /usr/lib/python2.7/dist- packages/nova/virt/libvirt/driver.py file we checked the share storage is true or not before migrating. def _is_storage_shared_with(self, dest, inst_base): # NOTE (rmk): There are two methods of determining whether we are # on the same filesystem: the source and dest IP are the # same, or we create a file on the dest system via SSH # and check whether the source system can also see it. shared_storage = (dest == self.get_host_ip_addr()) if not shared_storage: tmp_file = uuid.uuid4().hex + '.tmp' tmp_path = os.path.join(inst_base, tmp_file) LOG.debug("Temp path is") LOG.debug(tmp_path) try: utils.execute('ssh', dest, 'touch', tmp_path) if os.path.exists(tmp_path): shared_storage = True os.unlink(tmp_path) else: utils.execute('ssh', dest, 'rm', tmp_path) except Exception: pass return shared_storage But in case of share storage between compute node “shared storage = (dest == self.get_host_ip_addr())” always return false and also ssh mkdir return false and Nova resize will fail.I think checking the share storage using " shared_storage = (dest == self.get_host_ip_addr())" is not the right way. When I always return true from _is_storage_shared_with function, the Nova resize works fine. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1507521/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp