Public bug reported: Nova API removed onSharedStorage parameter after v2.14, but we still have many users use API under v2.14. And we found that nova 100% fail to evacuate a server via novaclient, the error log as follows:
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher [req-b34b79d6-20ed-4488-b8d5-98f4007eb12e 681041a7364d4852930021d009c8dc2b bafaf53fac4346b9bcd6a77cf964b7af - - -] Exception during message handling: Invalid state of instance files on shared storage 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher payload) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2809, in rebuild_instance 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2853, in _do_rebuild_instance_with_claim 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2885, in _do_rebuild_instance 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher _("Invalid state of instance files on shared" 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher I dug into the API source, and I found that the onSharedStorage will force to be converted to False value, even if in fact its value is None: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/evacuate.py#L43-L48 In other word, If we didn't pass this arguments, we will actually pass the False value. If we use shared-storage, it will not match the value getting from the backend driver. ** Affects: nova Importance: Undecided Status: New ** Description changed: - Nova API removed onSharedStorage parameter after v2.14, but we have many - user to API under v2.14. And we found that nova 100% fail to evacuate a - server via novaclient, the error log as follows: + Nova API removed onSharedStorage parameter after v2.14, but we still + have many users use API under v2.14. And we found that nova 100% fail to + evacuate a server via novaclient, the error log as follows: 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher [req-b34b79d6-20ed-4488-b8d5-98f4007eb12e 681041a7364d4852930021d009c8dc2b bafaf53fac4346b9bcd6a77cf964b7af - - -] Exception during message handling: Invalid state of instance files on shared storage 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher payload) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2809, in rebuild_instance 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2853, in _do_rebuild_instance_with_claim 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2885, in _do_rebuild_instance 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher _("Invalid state of instance files on shared" 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher - I dig into the API source, and I found that the onSharedStorage will - force to be converted to False, even if in fact its value is None: + I dug into the API source, and I found that the onSharedStorage will + force to be converted to False value, even if in fact its value is None: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/evacuate.py#L43-L48 - In other words, If we didn't pass this arguments, we will actually pass + In other word, If we didn't pass this arguments, we will actually pass the False value. If we use shared-storage, it will not match the value - from backend driver. + getting from the backend driver. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1665548 Title: nova fail to evacuate as wrong onSharedStorarge parameter Status in OpenStack Compute (nova): New Bug description: Nova API removed onSharedStorage parameter after v2.14, but we still have many users use API under v2.14. And we found that nova 100% fail to evacuate a server via novaclient, the error log as follows: 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher [req-b34b79d6-20ed-4488-b8d5-98f4007eb12e 681041a7364d4852930021d009c8dc2b bafaf53fac4346b9bcd6a77cf964b7af - - -] Exception during message handling: Invalid state of instance files on shared storage 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher payload) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2809, in rebuild_instance 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2853, in _do_rebuild_instance_with_claim 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs) 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2885, in _do_rebuild_instance 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher _("Invalid state of instance files on shared" 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage 2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher I dug into the API source, and I found that the onSharedStorage will force to be converted to False value, even if in fact its value is None: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/evacuate.py#L43-L48 In other word, If we didn't pass this arguments, we will actually pass the False value. If we use shared-storage, it will not match the value getting from the backend driver. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1665548/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp