* zhanghailiang (zhang.zhanghaili...@huawei.com) wrote: > This is the 3th version of COLO, it is only COLO frame part, include: VM > checkpoint, > failover, proxy API, block replication API, not include block replication. > The block part has been sent by wencongyang: > '[RFC PATCH 00/14] Block replication for continuous checkpoints' > > You can get the integrated qemu colo patches from github: > https://github.com/coloft/qemu/commits/colo-v1.0 > > Compared with the previous version, we have realized all parts of COLO frame, > and it is works now. > > The main change since last version is, we use colo proxy mode instead of > colo agent, they are all used for network packets compare, but proxy is more > efficient, it is based on netfilter. > Another modification is we implement new block replication scheme, > you can get more info from wencongyang's block patch series > > If you don't know about COLO, please refer to below link for detailed > information. > > The idea is presented in Xen summit 2012, and 2013, > and academia paper in SOCC 2013. It's also presented in KVM forum in 2013: > http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf > > Previous posted RFC proposal: > http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html > http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg04459.html > > The below is the detail about test COLO, you can also get this info > from http://wiki.qemu.org/Features/COLO. > * Hardware requirements > There is at least one directly connected nic to forward the network requests > from client to secondary VM. The directly connected nic must not be used by > any other purpose. > > * Network link topology > =================================normal ====================================== > +--------+ > |client | > master +----+---+ slave > -------------------------+ | + -------------------------+ > PVM | + | | > +-------+ +----[eth0]-----[switch]-----[eth0]---------+ | > |guest | +---+-+ | | +---+-+ | > | [tap0]--+ br0 | | | | br0 | | > | | +-----+ [eth1]-----[forward]----[eth1]--+ +-----+ SVM | > +-------+ | | | +-------+| > | | | +-----+ | guest || > [eth2]---[checkpoint]---[eth2] +--+br1 |-[tap0] || > | | +-----+ | || > | | +-------+| > -------------------------+ +--------------------------+ > e.g. > master: > br0: 192.168.0.33 > eth1: 192.168.1.33 > eth2: 192.168.2.33 > > slave: > br0: 192.168.0.88 > br1: no ip address > eth1: 192.168.1.88 > eth2: 192.168.2.88 > (Actually, you can also use eth0 as checkpoint channel) > Note: in normal, SVM will always be linked to br1 like above until > failover. > > * Test environment prepare: > 1. Set Up the Bridge and network environment > You must setup you network environment like above picture, > In master, setup a bridge br0, using command brctl, like: > # ifconfig eth0 down > # ifconfig eth0 0.0.0.0 > # brctl addbr br0 > # brctl addif br0 eth0 > # ifconfig br0 192.168.0.33 netmask 255.255.255.0 > # ifconfig eth0 up > In slave, setup two bridge br0, br1, commands are same with above, > please note that br1 is linked to eth1(the forward nic). > > 2.Qemu-ifup > We need a script to bring up the TAP interface. > You can find this info from http://en.wikibooks.org/wiki/QEMU/Networking. > Master: > root@master# cat /etc/qemu-ifup > #!/bin/sh > switch=br0 > if [ -n "$1" ]; then > ip link set $1 up > brctl addif ${switch} $1 > fi > Slave: > root@slave # cat /etc/qemu-ifup > #!/bin/sh > switch=br1 #in primary, switch is br0. in secondary switch is br1 > if [ -n "$1" ]; then > ip link set $1 up > brctl addif ${switch} $1 > fi > > 3. Prepare host kernel > colo-proxy kernel module need cooperate with linux kernel. > You should put a kernel patch 'colo-patch-for-kernel.patch' > (It's based on linux kernel-3.19) which you can get from > https://github.com/gao-feng/colo-proxy.git > and then compile kernel and intall the new kernel. > > 4. Proxy module > proxy module is used for network packets compare, you can also get the lastest > version from: https://github.com/gao-feng/colo-proxy.git. > You can compile and install it by using command 'make' && 'make install'. > > 5. Modified iptables > We have add a new rule to iptables command, so please get the patch from > https://github.com/gao-feng/colo-proxy/blob/master/COLO-library_for_iptables-1.4.21.patch > It is based on version 1.4.21.
I'm getting closer but I don't think I'm getting packets from the secondary to the primary yet; it looks like the primary is holding onto it's packets until the end of the 10second checkpoint. Still debugging that but I'd take any tips. > 2. startup qemu > master: > # qemu-system-x86_64 -enable-kvm -netdev > tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 > -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive > driver=quorum,read-pattern=first,children.0.file.filename=suse11_3.img,children.0.driver=raw,children.1.file.driver=nbd+colo,children.1.file.host=192.168.2.88,children.1.file.port=8889,children.1.file.export=colo1,children.1.driver=raw,if=virtio > -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor > stdio -S > slave: > # qemu-system-x86_64 -enable-kvm -netdev > tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 > -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive > driver=blkcolo,export=colo1,backing.file.filename=suse11_3.img,backing.driver=raw,if=virtio > -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor > stdio -incoming tcp:0:8888 > > 3. On Secondary VM's QEMU monitor, run > (qemu) nbd_server_start 192.168.2.88:8889 > > 4.on Primary VM's QEMU monitor, run following command: > (qemu) migrate_set_capability colo on > (qemu) migrate tcp:192.168.2.88:8888 > > 5. done > You will see two runing VMs, whenever you make changes to PVM, SVM > will be synced to PVM's state. > > 6. failover test: > You can kill SVM (PVM) and run 'colo_lost_heartbeat' in PVM's (SVM's) monitor > at the same time, then PVM (SVM) will failover and client will not feel this > change. I've not got that to work yet; so far if I kill the PVM the SVM quits shortly after. (I've tried both 'q' in the PVMs monitor and also kill -9). Dave > It is still a framework, far away from commercial use, > so any comments/feedbacks are warmly welcomed ;) > > PS: > We (huawei) have cooperated with fujitsu on COLO work, > and we work mainly on COLO frame and fujitsu will focus on COLO block. > > TODO list: > 1) Optimize the process of checkpoint, shorten the time-consuming > 2) Add more debug/stat info > 3) Strengthen failover > 4) The capability of continuous FT > > v3: > - use proxy instead of colo agent to compare network packets > - add block replication > - Optimize failover disposal > - handle shutdown > > v2: > - use QEMUSizedBuffer/QEMUFile as COLO buffer > - colo support is enabled by default > - add nic replication support > - addressed comments from Eric Blake and Dr. David Alan Gilbert > > v1: > - implement the frame of colo > > > zhanghailiang (27): > configure: Add parameter for configure to enable/disable COLO support > migration: Introduce capability 'colo' to migration > COLO: migrate colo related info to slave > migration: Integrate COLO checkpoint process into migration > migration: Integrate COLO checkpoint process into loadvm > migration: Don't send vm description in COLO mode > COLO: Implement colo checkpoint protocol > COLO: Add a new RunState RUN_STATE_COLO > QEMUSizedBuffer: Introduce two help functions for qsb > COLO: Save VM state to slave when do checkpoint > COLO RAM: Load PVM's dirty page into SVM's RAM cache temporarily > COLO VMstate: Load VM state into qsb before restore it > COLO RAM: Flush cached RAM into SVM's memory > COLO failover: Introduce a new command to trigger a failover > COLO failover: Implement COLO master/slave failover work > COLO failover: Don't do failover during loading VM's state > COLO: Add new command parameter 'colo_nicname' 'colo_script' for net > COLO NIC: Init/remove colo nic devices when add/cleanup tap devices > COLO NIC: Implement colo nic device interface configure() > COLO NIC : Implement colo nic init/destroy function > COLO NIC: Some init work related with proxy module > COLO: Do checkpoint according to the result of net packets comparing > COLO: Improve checkpoint efficiency by do additional periodic > checkpoint > COLO NIC: Implement NIC checkpoint and failover > COLO: Disable qdev hotplug when VM is in COLO mode > COLO: Implement shutdown checkpoint > COLO: Add block replication into colo process > > arch_init.c | 196 ++++++++- > configure | 14 + > hmp-commands.hx | 15 + > hmp.c | 7 + > hmp.h | 1 + > include/exec/cpu-all.h | 1 + > include/migration/migration-colo.h | 57 +++ > include/migration/migration-failover.h | 22 + > include/migration/migration.h | 14 + > include/migration/qemu-file.h | 3 +- > include/net/colo-nic.h | 25 ++ > include/net/net.h | 4 + > include/sysemu/sysemu.h | 3 + > migration/Makefile.objs | 2 + > migration/colo-comm.c | 81 ++++ > migration/colo-failover.c | 48 +++ > migration/colo.c | 743 > +++++++++++++++++++++++++++++++++ > migration/migration.c | 74 +++- > migration/qemu-file-buf.c | 57 +++ > net/Makefile.objs | 1 + > net/colo-nic.c | 438 +++++++++++++++++++ > net/tap.c | 45 +- > qapi-schema.json | 27 +- > qemu-options.hx | 10 +- > qmp-commands.hx | 19 + > savevm.c | 10 +- > scripts/colo-proxy-script.sh | 88 ++++ > stubs/Makefile.objs | 1 + > stubs/migration-colo.c | 49 +++ > vl.c | 36 +- > 30 files changed, 2047 insertions(+), 44 deletions(-) > create mode 100644 include/migration/migration-colo.h > create mode 100644 include/migration/migration-failover.h > create mode 100644 include/net/colo-nic.h > create mode 100644 migration/colo-comm.c > create mode 100644 migration/colo-failover.c > create mode 100644 migration/colo.c > create mode 100644 net/colo-nic.c > create mode 100755 scripts/colo-proxy-script.sh > create mode 100644 stubs/migration-colo.c > > -- > 1.7.12.4 > > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
