On Fri, Jul 06, 2018 at 11:56:59AM +0100, Dr. David Alan Gilbert wrote: > * Dr. David Alan Gilbert (dgilb...@redhat.com) wrote: > > * Peter Xu (pet...@redhat.com) wrote: > > > Based-on: <20180627132246.5576-1-pet...@redhat.com> > > > > > > Based on the series to unbreak postcopy: > > > Subject: [PATCH v3 0/4] migation: unbreak postcopy recovery > > > Message-Id: <20180627132246.5576-1-pet...@redhat.com> > > > > > > This series introduce a new postcopy recovery test. The new test > > > actually helped me to identify two bugs there so fix them as well > > > before 3.0 release. > > > > > > Patch 1: a trivial cleanup for existing postcopy ram load, which I > > > found a bit confusing during debugging the problem. > > > > > > Patch 2-3: two bug fixes that address different issues. Please see > > > the commit log for more information. > > > > > > Patch 4-9: add the postcopy recovery unit test. > > > > > > Please review. Thanks, > > > > Queued > > Hi Peter, > There's a problem in there somewhere; I'm getting > an intermittent failure of the test if I run a make check -j 8 on my > laptop. Just running two copies of tests/migration-test in parallel > sometimes triggers it (but not if I turn on QTEST_LOG!). > But it's always failing with: > > > ERROR:/home/dgilbert/git/migpull/tests/migration-test.c:373:migrate_recover: > assertion failed: (qdict_haskey(rsp, "return")) > > Dave
Hi Peter, Dave, I have applied this patchset in upstream Qemu to test postcopy pause/recovery. I observed error after triggering recovery command from source monitor where the target is lost and the source remains to be in `postcopy-pause` state. Please find my observation below, Source: # ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none -machine \ pseries -m 64G,slots=128,maxmem=128G -smp 16,maxcpus=32 -device virtio-blk-pci,drive=rootdisk \ -drive file=/home/hostos-ppc64le.qcow2,if=none,cache=none,format=qcow2,id=rootdisk \ -monitor telnet:127.0.0.1:1234,server,nowait -net nic,model=virtio -net user \ -redir tcp:2000::22 qemu-system-ppc64: Detected IO failure for postcopy. Migration paused. Source Monitor: (qemu) migrate_set_capability postcopy-ram on (qemu) migrate_set_parameter max-postcopy-bandwidth 4096 (qemu) migrate -d tcp:127.0.0.1:4444 (qemu) migrate_start_postcopy (qemu) migrate_pause (qemu) migrate -r tcp:127.0.0.1:4446 After triggering recovery, target is lost with the error mentioned below and source remains to be in `postcopy-paused` state (qemu) info migrate globals: store-global-state: on only-migratable: off send-configuration: on send-section-footer: on decompress-error-check: on capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off \ compress: off events: off postcopy-ram: on x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off \ dirty-bitmaps: off postcopy-blocktime: off late-block-activate: off Migration status: postcopy-recover total time: 78818 milliseconds expected downtime: 300 milliseconds setup: 169 milliseconds transferred ram: 177749 kbytes throughput: 63.72 mbps remaining ram: 28061376 kbytes total ram: 67109120 kbytes duplicate: 9742102 pages skipped: 0 pages normal: 22986 pages normal bytes: 91944 kbytes dirty sync count: 2 page size: 4 kbytes multifd bytes: 0 kbytes dirty pages rate: 1273187 pages postcopy request count: 236 Target: # ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none -machine \ pseries -m 64G,slots=128,maxmem=128G -smp 16,maxcpus=32 -device virtio-blk-pci,drive=rootdisk \ -drive file=/home/bala/sharing/hostos-ppc64le.qcow2,if=none,cache=none,format=qcow2,id=rootdisk \ -monitor telnet:127.0.0.1:1235,server,nowait -net nic,model=virtio -net user \ -redir tcp:2001::22 -incoming tcp:127.0.0.1:4444 qemu-system-ppc64: check_section_footer: Read section footer failed: -5 qemu-system-ppc64: Detected IO failure for postcopy. Migration paused. qemu-system-ppc64: Not a migration stream qemu-system-ppc64: load of migration failed: Invalid argument Target Monitor: (qemu) migrate_set_capability postcopy-ram on (qemu) migrate_recover tcp:127.0.0.1:4446 (qemu) Connection closed by foreign host. QTest: Also with respect to Qtest, I have tested it and the recovery test doesn't complete as it waits on the source for "completed" but due to this issue source remains to be in `postcopy-paused` `migrate_postcopy_complete(from, to);` but it actually doesn't end. As it did not complete, I cancelled it forcefully # time QTEST_QEMU_BINARY=./ppc64-softmmu/qemu-system-ppc64 ./tests/migration-test /ppc64/migration/deprecated: OK /ppc64/migration/bad_dest: OK /ppc64/migration/postcopy/unix: OK /ppc64/migration/postcopy/recovery: ^C real 21m55.176s user 2m28.800s sys 4m55.980s -- Bala > > > > Peter Xu (9): > > > migration: simplify check to use qemu file buffer > > > migration: loosen recovery check when load vm > > > migration: fix incorrect bitmap size calculation > > > tests: introduce migrate_postcopy_* helpers > > > tests: allow migrate() to take extra flags > > > tests: introduce migrate_query*() helpers > > > tests: introduce wait_for_migration_status() > > > tests: add postcopy recovery test > > > tests: hide stderr for postcopy recovery test > > > > > > migration/ram.c | 21 +++-- > > > migration/savevm.c | 16 ++-- > > > tests/migration-test.c | 198 ++++++++++++++++++++++++++++++++--------- > > > 3 files changed, 176 insertions(+), 59 deletions(-) > > > > > > -- > > > 2.17.1 > > > > > > > > -- > > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK >
