Reviewed: https://review.opendev.org/c/openstack/nova/+/801714
Committed:
https://opendev.org/openstack/nova/commit/49ba5a763f9d2c7c496f391ca8149a18541bfec7
Submitter: "Zuul (22348)"
Branch: master
commit 49ba5a763f9d2c7c496f391ca8149a18541bfec7
Author: Lee Yarwood <lyarw...@redhat.com>
Date: Thu Jul 22 09:05:51 2021 +0100
libvirt: Handle silent failures to extend volume within os-brick
As seen in bug #1849425 os-brick can fail to extend an underlying volume
device on the compute silently, returning a new_size of None to n-cpu.
While this should ultimatley be addressed in os-brick n-cpu can also
handle this before we eventually run into a type error when attemting
floor division later in the volume extend flow.
Change-Id: Ic8091537274a5ad27fb5af8939f81ed154b7ad7c
Closes-Bug: #1849425
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1849425
Title:
Extend attached volume case failed
Status in Cinder:
New
Status in OpenStack Compute (nova):
Fix Released
Bug description:
The online expansion volume use case failed,
tempest.api.volume.test_volumes_extend:VolumesExtendAttachedTest.test_extend_attached_volume.
This use case was originally successfully executed and failed after
reinstalling devstack (stein-->train). The cinder driver has not
changed, and the multipath.conf configuration has not changed. There
are two kinds of errors in devstack@n-cpu.service, one is the error
under the ubuntu18 system, and the other is the error under the
centos7 system. as follows
ubuntu18:
Oct 23 09:34:52 stack nova-compute[11706]: WARNING nova.compute.manager
[req-0ce8f571-5c04-4a72-8049-3839c934c7c9
req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] [instance:
6a4b5b05-20b7-46c1-b903-b6fe40a38aad] Extend volume failed,
volume_id=bc3c5af6-b6c6-4e01-8d0e-774ad637debc, reason: unsupported operand
type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
[req-0ce8f571-5c04-4a72-8049-3839c934c7c9
req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] Exception during message
handling: TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
Traceback (most recent call last):
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
res = self.dispatcher.dispatch(message)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 265, in dispatch
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
return self._do_dispatch(endpoint, method, ctxt, args)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 194, in _do_dispatch
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
result = func(ctxt, **new_args)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
function_name, call_dict, binary, tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220,
in __exit__
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196,
in force_reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
raise value
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
return f(self, context, *args, **kw)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 8587, in
external_instance_event
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
self.extend_volume(context, instance, event.tag)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
return function(self, context, *args, **kwargs)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220,
in __exit__
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196,
in force_reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
raise value
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
return function(self, context, *args, **kwargs)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
bdm.volume_size * units.Gi)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1768, in extend_volume
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
dev.resize(new_size // units.Ki)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server
centos7:
Oct 10 10:15:12 devstack nova-compute[59879]: DEBUG
os_brick.initiator.connectors.iscsi [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed
req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] <== extend_volume:
exception (371ms) ProcessExecutionError('fail\n', '', 1, 'multipathd resize map
3600b34245364482d2d43d02ffda900df', None) {{(pid=59879) trace_logging_wrapper
/usr/local/lib/python3.7/site-packages/os_brick/utils.py:156}}
Oct 10 10:15:12 devstack nova-compute[59879]: WARNING nova.compute.manager
[req-fbe8d107-47e2-44b5-af76-413b6e6a24ed
req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] [instance:
72d1a0fc-91bf-46eb-9ef8-96a81886a747] Extend volume failed,
volume_id=79481f44-72dd-4c07-98cb-70f52048f10e, reason: Unexpected error while
running command.
Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map
3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: '':
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
[req-fbe8d107-47e2-44b5-af76-413b6e6a24ed
req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] Exception during message
handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error
while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map
3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: ''
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
Traceback (most recent call last):
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py",
line 166, in _process_incoming
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
res = self.dispatcher.dispatch(message)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File
"/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
265, in dispatch
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return self._do_dispatch(endpoint, method, ctxt, args)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File
"/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
194, in _do_dispatch
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
result = func(ctxt, **new_args)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
function_name, call_dict, binary, tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
raise value
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return f(self, context, *args, **kw)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 8587, in
external_instance_event
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
self.extend_volume(context, instance, event.tag)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return function(self, context, *args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 215, in
decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
raise value
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 203, in
decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return function(self, context, *args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
bdm.volume_size * units.Gi)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1738, in
extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
requested_size)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1371, in
_extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
requested_size)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 86, in
extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
new_size = self.connector.extend_volume(connection_info['data'])
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/os_brick/utils.py", line 150, in
trace_logging_wrapper
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
result = f(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_concurrency/lockutils.py",
line 328, in inner
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return f(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File
"/usr/local/lib/python3.7/site-packages/os_brick/initiator/connectors/iscsi.py",
line 477, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return self._linuxscsi.extend_volume(volume_paths)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File
"/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line
572, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
result = self.multipath_resize_map(scsi_wwn)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File
"/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line
531, in multipath_resize_map
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
root_helper=self._root_helper)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/os_brick/executor.py", line 52,
in _execute
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
result = self.__execute(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File
"/usr/local/lib/python3.7/site-packages/os_brick/privileged/rootwrap.py", line
169, in execute
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return execute_root(*cmd, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_privsep/priv_context.py",
line 241, in _wrap
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
return self.channel.remote_call(name, args, kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
File "/usr/local/lib/python3.7/site-packages/oslo_privsep/daemon.py", line
203, in remote_call
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
raise exc_type(*result[2])
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
Stderr: ''
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server
These two errors will be resolved after changing user_friendly_names in
multipath.conf to no. What is the problem with this?
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1849425/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
