Hello everyone, I am seeing a strange issue with cinder block live migration
and libvirt and looking for some assistance.
Environment: Openstack Pike
OS: Ubuntu 16.04LTS
Cinder FC Driver: Pure Storage
Cinder FC Driver: Dell Compellent
libvirtd (libvirt) 3.6.0
Cinder-volume 2:11.1.0-0ubuntu1~cloud0
I am trying to migrate some volumes from our older Dell Compellent SC storage
arrays to new Pure Storage arrays. I can create new volumes on both array’s
via cinder. I can create a volume and boot it on the Pure AFA’s. I can live
migrate volumes from the Pure array to the Compellent array but not from the
Compellent to the Pure. The volume is created and data copied to it, it is then
deleted by Cinder after the failure is logged. I receive the following error
when going from the Compellent to the Pure Array. I can migrate in both
directions if the volume is not attached to an instance. Any ideas?
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver
[req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Failure rebasing volume
/dev/disk/by-path/pci-0000:03:00.0-fc-0x524a937cddfa5902-lun-2 on vda.:
libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an
active copy job
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver Traceback (most
recent call last):
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1345, in
_swap_volume
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver
dev.abort_job(pivot=True)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 751, in
abort_job
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver
self._guest._domain.blockJobAbort(self._disk, flags=flags)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver result =
proxy_call(self._autowrap, f, *args, **kwargs)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv = execute(f,
*args, **kwargs)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver six.reraise(c,
e, tb)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv =
meth(*args, **kwargs)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File
"/usr/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver if ret == -1:
raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver libvirtError:
Requested operation is not valid: pivot of disk 'vda' requires an active copy
job
2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager
[req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] Failed to swap volume
bb59023e-d463-44e9-8b1a-a9af495d3d4f for ecc059d0-79f9-402e-aea8-8c99004f221d:
VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid:
pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] Traceback (most recent call last):
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in
_swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] mountpoint, resize_to)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in
swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] instance)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] self.force_reraise()
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] six.reraise(self.type_, self.value,
self.tb)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in
swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] self._swap_volume(guest, disk_dev,
conf, resize_to)
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in
_swap_volume
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] raise
exception.VolumeRebaseFailed(reason=six.text_type(exc))
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29] VolumeRebaseFailed: Volume rebase failed:
Requested operation is not valid: pivot of disk 'vda' requires an active copy
job
2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance:
f834fc03-7e2d-41f4-9307-a1bded3abb29]
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
[req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Exception during message
handling: VolumeRebaseFailed: Volume rebase failed: Requested operation is not
valid: pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in
_process_incoming
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213,
in dispatch
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183,
in _do_dispatch
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in
wrapped
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server function_name,
call_dict, binary)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in
wrapped
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 218, in
decorated_function
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 206, in
decorated_function
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5255, in
swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
is_cinder_migration)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5154, in
_swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server context,
new_attachment_id)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in
_swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server mountpoint,
resize_to)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in
swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server instance)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
self.force_reraise()
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in
swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
self._swap_volume(guest, disk_dev, conf, resize_to)
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in
_swap_volume
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server raise
exception.VolumeRebaseFailed(reason=six.text_type(exc))
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid:
pivot of disk 'vda' requires an active copy job
2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server
Any help would be greatly appreciated.
Thanks!
— Steve
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
