Hi All,
it asked this on linux-ha but had no luck. Is anybody here who has some
hints for me or could tell me if it is possible (and sensible) to live
migrate a virtual machine if the disk image is provided by a multipath
device?
I am not sure if my approach is flawed or if I am using the wrong or
misconfigured tools.
Thanks a lot and sorry for crossposting,
Sven
Dear All,
I have set up a three node cluster with shared storage (DRBD
active/passive) which exports iSCSI Volumes (TGT) containing KVM/QEMU
disk images.
The iSCSI Target is configured as one resource and accessible on two
floating ip addresses to allow multipath I/O for speed and redundancy.
The VM hosts are accessing the volumes via open-isci using dm-multipath
(grouping_policy multibus).
While migrating the iSCSI Target from A to B everything works fine.
But if I try to live migrate a virtual machine I experience file system
corruptions inside the virtual machine. So, somehow the switching of the
iSCSI/Multipath Sessions is not handled properly by the VM hosts.
I have configured iSCSI timeouts rather short (noop_out_timeout 5
seconds) and "no_path_retry queue" on the multipath device.
My question(s):
1) Is it conceptually wrong what I am trying to accomplish?
3) Is it valid to use "no_path_retry queue" in such a setup?
4) Did I miss some important configuration options (timings, etc.)?
5) Is TGT multipath capable?
Thank you all for any hints,
Sven
===== Additional Information below: =====
- Cluster Layout
- Environment
- multipath configuration
- iSCSI Timings
- cib configuration (simplified and sorted)
Cluster Layout:
---------------
A B C
(active) <--- DRBD ---> (passive)
iSCSI Target
ip0 ip1 -- failover-->
(floating IPs)
---------------- iSCSI Initiator -----------------------
(two pathes)
---------------- Multipath I/O -----------------------
---------------- libvirt/KVM -----------------------
<------------------------------ failover ----- VM1
Environment:
------------
Ubuntu 12.04.2 LTS
kernel 3.5.0.34
corosync 1.4.2-2
cman 3.1.7-0ubuntu2.1
pacemaker 1.1.6-2ubuntu3
resource-agents 1:3.9.2-5ubuntu4.1
tgt 1:1.0.17-1ubuntu2
open-iscsi 2.0.871-0ubuntu9.12.04.2
multipath-tools 0.4.9-3ubuntu5
multipath.conf:
---------------
defaults {
udev_dir /dev
polling_interval 10
path_selector "round-robin 0"
path_grouping_policy multibus
path_checker readsector0
rr_min_io 100
max_fds 8192
rr_weight priorities
failback immediate
no_path_retry queue
}
iscsi timeouts (from /etc/iscsid/iscsi.conf):
---------------------------------------------
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
cib configuration (excerpt, slightly modified):
-----------------------------------------------
primitive p-drbd-r0 ocf:linbit:drbd \
params drbd_resource="r0" \
op monitor interval="15"
ms ms-drbd-r0 p-drbd-r0 \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true"
primitive p-vg_drbd ocf:heartbeat:LVM \
params volgrpname="vg_drbd" \
op monitor interval="30s" timeout="30s" \
op start interval="0" timeout="30s" \
op stop interval="0" timeout="30s"
primitive p-iscsi-target ocf:heartbeat:iSCSITarget \
params iqn="<iscsi iqn>" tid="1" implementation="tgt"
allowed_initiators="<..initiator ips..>" \
op monitor interval="15s"
primitive p-lun1-vm_disk ocf:heartbeat:iSCSILogicalUnit \
params target_iqn="iqn.2013-03.de.localite:storage" lun="1"
path="/dev/vg_drbd/vm_disk" implementation="tgt" vendor_id="STGT"
primitive p-iscsiip0 ocf:heartbeat:IPaddr2 \
params ip="10.223.101.131" nic="eth2" cidr_netmask="26" \
op monitor interval="20s"
primitive p-iscsiip1 ocf:heartbeat:IPaddr2 \
params ip="10.223.101.195" nic="eth3" cidr_netmask="26" \
op monitor interval="20s"
group rg-iscsitarget p-iscsi-target p-lun1-vm_disk p-iscsiip0 p-iscsiip1
primitive p-iscsi-initiator lsb:open-iscsi \
op monitor interval="30s"
clone clone-iscsiinitiator p-iscsi-initiator \
meta interleave="true"
primitive p-libvirtd lsb:libvirt-bin \
op monitor interval="30s"
clone clone-libvirtd p-libvirtd \
meta interleave="true"
primitive p-vm ocf:heartbeat:VirtualDomain \
params config="/etc/libvirt/qemu/vm.xml"
migration_transport="tls" \
meta allow-migrate="true" \
op start interval="0" timeout="250s" \
op stop interval="0" timeout="300s" \
op monitor interval="60s" timeout="30s" \
op migrate_from interval="0" timeout="300s" \
op migrate_to interval="0" timeout="300s"
colocation col-iscsitarget_on_drbd inf: rg-iscsitarget ms-drbd-r0:Master
order o-drbd-r0_before_vg inf: ms-drbd-r0:promote p-vg_drbd:start
order o-vg-drbd-r0_before_iscsitarget inf: p-vg_drbd rg-iscsitarget
order o-iscsitarget_before_iscsiinitiator 0: rg-iscsitarget
clone-iscsiinitiator
order o-iscsiinitiator_before_libvirt 0: clone-iscsiinitiator clone-libvirtd
order o-libvirt_before_vm inf: clone-libvirtd p-vm
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
