Hello,
I'm trying to live-migrate a volume-based instance and it fails. I have
successfully done this in Havana, but Icehouse either works differently
or I'm missing something.
I'm using Centos6.5 with KVM, local firewalls and selinux are disabled
on all nodes.
I have a controller, 3 compute nodes (compute1 to compute3) and 3 block
storage nodes (block1 to block3).
I'm creating an instance using "boot-from-image-creates-new-volume"; the
instance gets created on compute2 using a volume created on block2 and
works fine, although I can see
2014-05-21 16:31:48.487 1462 WARNING nova.virt.libvirt.driver
[req-e53e2353-2073-4a09-8d3b-94298a982e2d
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478]
[instance: a20aa279-4701-47c7-9be1-ae5846a00d91] File injection into a
boot from volume instance is not supported
I have made the modifications to libvirtd.conf, nova.conf and
/etc/sysconfig/libvirtd suggested here:
http://kimizhang.wordpress.com/2013/08/26/openstack-vm-live-migration/
From the CLI, I do
nova live-migration a20aa279-4701-47c7-9be1-ae5846a00d91 compute3.gbif.clu
With havana this used to work; with Icehouse, the instance ends up on
compute3, with status 'Active' but power state is 'Paused'. No errors in
the compute or libvirtd logs of source compute node, but on destination
(compute3) comput.log says:
2014-05-21 17:00:37.135 1467 WARNING nova.compute.manager [-] [instance:
a20aa279-4701-47c7-9be1-ae5846a00d91] Instance is paused unexpectedly.
Ignore.
Instance stopped with 'nova stop a20aa279-4701-47c7-9be1-ae5846a00d91'.
Trying to start with 'instance start' results in the following erros in
libvirtd.log on compute3:
2014-05-22 09:53:19.396+0000: 1443: error : virFileOpenAs:1093 : failed
to open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
2014-05-22 09:53:19.396+0000: 1443: error :
virStorageFileGetMetadataRecurse:939 : cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
with corresponding errors in compute3's compute.log (see below).
The thing is, on compute3, we have:
[root@compute3 nova]# ls -alh
/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1
lrwxrwxrwx 1 root root 9 May 22 11:49
/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1
-> ../../sdb
so the symlink exists....
'Regular' migration works on the installation with this type of volume
based instances.
I was wondering if anyone could shed any light on this.
Thanks!
Best regards,
Andrei
Compute.log full errors:
2014-05-22 11:53:17.668 1467 ERROR glanceclient.common.http [-] Request
returned failure status.
2014-05-22 11:53:17.668 1467 WARNING nova.compute.utils
[req-5c09ada2-2072-4858-91a7-063b3d6224ed
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478]
[instance: a20aa279-4701-47c7-9be1-ae5846a00d91] Can't access image :
Image could not be found.
2014-05-22 11:53:19.398 1467 ERROR nova.virt.libvirt.driver
[req-5c09ada2-2072-4858-91a7-063b3d6224ed
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478] An
error occurred while trying to launch a defined domain with xml: <domain
type='kvm'>
<name>instance-00000015</name>
<uuid>a20aa279-4701-47c7-9be1-ae5846a00d91</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Red Hat Inc.</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2014.1-2.el6</entry>
<entry name='serial'>96ccf268-2a3e-11dc-a7d0-000ea68f7399</entry>
<entry name='uuid'>a20aa279-4701-47c7-9be1-ae5846a00d91</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='rhel6.5.0'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source
dev='/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1'/>
<target dev='vda' bus='virtio'/>
<serial>b897a227-e7ca-4179-a17d-3e00700557a4</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<interface type='bridge'>
<mac address='fa:16:3e:78:0f:fa'/>
<source bridge='br100'/>
<model type='virtio'/>
<filterref filter='nova-instance-instance-00000015-fa163e780ffa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='file'>
<source
path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
<target port='0'/>
</serial>
<serial type='pty'>
<target port='1'/>
</serial>
<console type='file'>
<source
path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'
keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>
2014-05-22 11:53:19.896 1467 ERROR oslo.messaging.rpc.dispatcher [-]
Exception during message handling: cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
Traceback (most recent call last):
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 133, in _dispatch_and_reply
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 176, in _dispatch
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 122, in _do_dispatch
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher result
= getattr(endpoint, method)(ctxt, **new_args)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher payload)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280, in
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher pass
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266, in
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333, in
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 309, in
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296, in
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300,
in start_instance
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
self._power_on(context, instance)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288,
in _power_on
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
block_device_info)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2128, in power_on
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
self._hard_reboot(context, instance, network_info, block_device_info)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2092, in _hard_reboot
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
vifs_already_plugged=True)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3644, in _create_domain_and_network
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
power_on=power_on)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3547, in _create_domain
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
domain.XMLDesc(0))
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3542, in _create_domain
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
domain.createWithFlags(launch_flags)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher result
= proxy_call(self._autowrap, f, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in
proxy_call
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv =
execute(f,*args,**kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv =
meth(*args,**kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in
createWithFlags
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher if ret
== -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
libvirtError: cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-]
Returning exception cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address to caller
2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-]
['Traceback (most recent call last):\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 133, in _dispatch_and_reply\n incoming.message))\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 176, in _dispatch\n return self._do_dispatch(endpoint, method,
ctxt, args)\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
**new_args)\n', ' File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in
wrapped\n payload)\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in
wrapped\n return f(self, context, *args, **kw)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280, in
decorated_function\n pass\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266, in
decorated_function\n return function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333, in
decorated_function\n function(self, context, *args, **kwargs)\n', '
File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line
309, in decorated_function\n e, sys.exc_info())\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296, in
decorated_function\n return function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300,
in start_instance\n self._power_on(context, instance)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288,
in _power_on\n block_device_info)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2128, in power_on\n self._hard_reboot(context, instance,
network_info, block_device_info)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2092, in _hard_reboot\n vifs_already_plugged=True)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3644, in _create_domain_and_network\n power_on=power_on)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3547, in _create_domain\n domain.XMLDesc(0))\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3542, in _create_domain\n domain.createWithFlags(launch_flags)\n', '
File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in
doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '
File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in
proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in
tworker\n rv = meth(*args,**kwargs)\n', ' File
"/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in
createWithFlags\n if ret == -1: raise libvirtError
(\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address\n"]
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
