** Changed in: nova
Status: In Progress => Fix Released
** Changed in: nova
Importance: Undecided => High
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1861071
Title:
Failing to extend an attached encrypted volume
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Description:
While extending an attached encrypted volume, Cinder reports volume had been
extended.
However inside the instance lsblk doesn't show new extended size.
This works fine for none encrypted attached volumes.
The error on Nova says suggest this isn't cause by Cinder or it's
backend thus also Cinder backend agnostic.
If this won't be fixed or supported, we should block or at the very
least document that it's a known limitation.
Steps to reproduce:
1. Create an encrypted volume say 1G
2. Attach said volume to instance
3. check lsblk inside instance should report ~1G
4. Cinder extend attached encrypted volume example, notice needs Cinder micro
version
cinder --os-volume-api-version 3.59 extend [VolID] 3
5. Recheck cinder show [VolID] notice volume size was extended
6. Inside instance lsblk doesn't show the new extended size.
7. Notice Nova error (seen later below)
Version:
openstack-nova-compute-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
openstack-nova-migration-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
python3-novaclient-15.1.0-0.20190919143437.cd396b8.el8ost.noarch
python3-nova-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
puppet-nova-15.4.1-0.20191126042922.b1bb388.el8ost.noarch
openstack-nova-common-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
Expected result:
lsblk should report new size, as it successfully does in case of extending an
attached none encrypted volume.
Actual result:
Cinder show volume size increase
But lsblk doesn't grow.
If not supported this should be blocked/alerted on logs.
Or at the very least documented.
Traceback from nova about failures in resize:
var/log/containers/nova/nova-compute.log:71:2020-01-28 07:14:17.394 7 ERROR
nova.virt.libvirt.driver [instance: f93e235e-30e9-4bdc-ad48-8e972552bbb5]
/var/log/containers/nova/nova-compute.log:73:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server [req-ef88174f-6762-4842-a64c-76074a848ac7
b753e5c55ba94950b5463ae41bb623ab 08d5a55b6cad4413910abc863b4a2b15 - default
default] Exception during message handling: libvirt.libvirtError: internal
error: unable to execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:74:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log:75:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
/var/log/containers/nova/nova-compute.log:76:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
/var/log/containers/nova/nova-compute.log:77:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274,
in dispatch
/var/log/containers/nova/nova-compute.log:78:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt,
args)
/var/log/containers/nova/nova-compute.log:79:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194,
in _do_dispatch
/var/log/containers/nova/nova-compute.log:80:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server result = func(ctxt, **new_args)
/var/log/containers/nova/nova-compute.log:81:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in
wrapped
/var/log/containers/nova/nova-compute.log:82:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server function_name, call_dict, binary, tb)
/var/log/containers/nova/nova-compute.log:83:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:84:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server self.force_reraise()
/var/log/containers/nova/nova-compute.log:85:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
/var/log/containers/nova/nova-compute.log:86:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:87:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py",
line 693, in reraise
/var/log/containers/nova/nova-compute.log:88:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server raise value
/var/log/containers/nova/nova-compute.log:89:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in
wrapped
/var/log/containers/nova/nova-compute.log:90:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server return f(self, context, *args, **kw)
/var/log/containers/nova/nova-compute.log:91:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9223, in
external_instance_event
/var/log/containers/nova/nova-compute.log:92:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server self.extend_volume(context, instance, event.tag)
/var/log/containers/nova/nova-compute.log:93:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in
decorated_function
/var/log/containers/nova/nova-compute.log:94:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:95:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in
decorated_function
/var/log/containers/nova/nova-compute.log:96:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
/var/log/containers/nova/nova-compute.log:97:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:98:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server self.force_reraise()
/var/log/containers/nova/nova-compute.log:99:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
/var/log/containers/nova/nova-compute.log:100:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:101:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py",
line 693, in reraise
/var/log/containers/nova/nova-compute.log:102:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server raise value
/var/log/containers/nova/nova-compute.log:103:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in
decorated_function
/var/log/containers/nova/nova-compute.log:104:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:105:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9081, in
extend_volume
/var/log/containers/nova/nova-compute.log:106:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server bdm.volume_size * units.Gi)
/var/log/containers/nova/nova-compute.log:107:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2048, in
extend_volume
/var/log/containers/nova/nova-compute.log:108:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server instance=instance)
/var/log/containers/nova/nova-compute.log:109:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:110:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server self.force_reraise()
/var/log/containers/nova/nova-compute.log:111:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
/var/log/containers/nova/nova-compute.log:112:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:113:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py",
line 693, in reraise
/var/log/containers/nova/nova-compute.log:114:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server raise value
/var/log/containers/nova/nova-compute.log:115:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2037, in
extend_volume
/var/log/containers/nova/nova-compute.log:116:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server dev.resize(new_size // units.Ki)
/var/log/containers/nova/nova-compute.log:117:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 809, in
resize
/var/log/containers/nova/nova-compute.log:118:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server self._guest._domain.blockResize(self._disk,
size_kb)
/var/log/containers/nova/nova-compute.log:119:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
/var/log/containers/nova/nova-compute.log:120:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args,
**kwargs)
/var/log/containers/nova/nova-compute.log:121:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
/var/log/containers/nova/nova-compute.log:122:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:123:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
/var/log/containers/nova/nova-compute.log:124:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server six.reraise(c, e, tb)
/var/log/containers/nova/nova-compute.log:125:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py",
line 693, in reraise
/var/log/containers/nova/nova-compute.log:126:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server raise value
/var/log/containers/nova/nova-compute.log:127:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
/var/log/containers/nova/nova-compute.log:128:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server rv = meth(*args, **kwargs)
/var/log/containers/nova/nova-compute.log:129:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server File
"/usr/lib64/python3.6/site-packages/libvirt.py", line 1006, in blockResize
/var/log/containers/nova/nova-compute.log:130:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server if ret == -1: raise libvirtError
('virDomainBlockResize() failed', dom=self)
/var/log/containers/nova/nova-compute.log:131:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server libvirt.libvirtError: internal error: unable to
execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:132:2020-01-28 07:14:17.778 7 ERROR
oslo_messaging.rpc.server
/var/log/containers/libvirt/libvirtd.log:2:2020-01-28 07:14:17.392+0000:
4425: error : qemuMonitorJSONCheckError:418 : internal error: unable to execute
QEMU command 'block_resize': Cannot grow device files
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1861071/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
