On Thu, Sep 12, 2024 at 09:13:16AM +0100, Peter Maydell wrote: > On Wed, 11 Sept 2024 at 22:26, Fabiano Rosas <faro...@suse.de> wrote: > > I don't think we're discussing total CI time at this point, so the math > > doesn't really add up. We're not looking into making the CI finish > > faster. We're looking into making migration-test finish faster. That > > would reduce timeouts in CI, speed-up make check and reduce the chance > > of random race conditions* affecting other people/staging runs. > > Right. The reason migration-test appears on my radar is because > it is very frequently the thing that shows up as "this sometimes > just fails or just times out and if you hit retry it goes away > again". That might not be migration-test's fault specifically, > because those retries tend to be certain CI configs (s390, > the i686-tci one), and I have some theories about what might be > causing it (e.g. build system runs 4 migration-tests in parallel, > which means 8 QEMU processes which is too many for the number > of host CPUs). But right now I look at CI job failures and my reaction > is "oh, it's the migration-test failing yet again" :-( > > For some examples from this week: > > https://gitlab.com/qemu-project/qemu/-/jobs/7802183144 > https://gitlab.com/qemu-project/qemu/-/jobs/7799842373 <--------[1] > https://gitlab.com/qemu-project/qemu/-/jobs/7786579152 <--------[2] > https://gitlab.com/qemu-project/qemu/-/jobs/7786579155
Ah right, the TIMEOUT is unfortunate, especially if tests can be run in parallel. It indeed sounds like no good way to finally solve.. I don't also see how speeding up / reducing tests in migration test would help, as that's (from some degree..) is the same as tuning the timeout value bigger. When the tests are less it'll fit into 480s window, but maybe it's too quick now we wonder whether we should shrink it to e.g. 90s, but then it can timeout again when on a busy host with less capability of concurrency. But indeed there're two ERRORs ([1,2] above).. I collected some more info here before the log expires: =================================8<================================ *** /i386/migration/multifd/tcp/plain/cancel, qtest-i386 on s390 host https://gitlab.com/qemu-project/qemu/-/jobs/7799842373 101/953 qemu:qtest+qtest-i386 / qtest-i386/migration-test ERROR 144.32s killed by signal 6 SIGABRT >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon >>> G_TEST_DBUS_DAEMON=/home/gitlab-runner/builds/zEr9wY_L/0/qemu-project/qemu/tests/dbus-vmstate-daemon.sh >>> >>> PYTHON=/home/gitlab-runner/builds/zEr9wY_L/0/qemu-project/qemu/build/pyvenv/bin/python3 >>> QTEST_QEMU_IMG=./qemu-img MALLOC_PERTURB_=144 >>> QTEST_QEMU_BINARY=./qemu-system-i386 >>> /home/gitlab-runner/builds/zEr9wY_L/0/qemu-project/qemu/build/tests/qtest/migration-test >>> --tap -k ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― stderr: warning: fd: migration to a file is deprecated. Use file: instead. warning: fd: migration to a file is deprecated. Use file: instead. ../tests/qtest/libqtest.c:205: kill_qemu() detected QEMU death from signal 11 (Segmentation fault) (core dumped) (test program exited with status code -6) TAP parsing error: Too few tests run (expected 53, got 39) ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― # Start of plain tests # Running /i386/migration/multifd/tcp/plain/cancel # Using machine type: pc-i440fx-9.2 # starting QEMU: exec ./qemu-system-i386 -qtest unix:/tmp/qtest-3273509.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-3273509.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -accel kvm -accel tcg -machine pc-i440fx-9.2, -name source,debug-threads=on -m 150M -serial file:/tmp/migration-test-4112T2/src_serial -drive if=none,id=d0,file=/tmp/migration-test-4112T2/bootsect,format=raw -device ide-hd,drive=d0,secs=1,cyls=1,heads=1 2>/dev/null -accel qtest # starting QEMU: exec ./qemu-system-i386 -qtest unix:/tmp/qtest-3273509.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-3273509.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -accel kvm -accel tcg -machine pc-i440fx-9.2, -name target,debug-threads=on -m 150M -serial file:/tmp/migration-test-4112T2/dest_serial -incoming defer -drive if=none,id=d0,file=/tmp/migration-test-4112T2/bootsect,format=raw -device ide-hd,drive=d0,secs=1,cyls=1,heads=1 2>/dev/null -accel qtest ----------------------------------- stderr ----------------------------------- warning: fd: migration to a file is deprecated. Use file: instead. warning: fd: migration to a file is deprecated. Use file: instead. ../tests/qtest/libqtest.c:205: kill_qemu() detected QEMU death from signal 11 (Segmentation fault) (core dumped) *** /ppc64/migration/multifd/tcp/plain/cancel, qtest-ppc64 on i686 host https://gitlab.com/qemu-project/qemu/-/jobs/7786579152 174/315 qemu:qtest+qtest-ppc64 / qtest-ppc64/migration-test ERROR 381.00s killed by signal 6 SIGABRT >>> PYTHON=/builds/qemu-project/qemu/build/pyvenv/bin/python3 >>> QTEST_QEMU_IMG=./qemu-img >>> G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh >>> QTEST_QEMU_BINARY=./qemu-system-ppc64 MALLOC_PERTURB_=178 >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon >>> /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap -k ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― stderr: qemu-system-ppc64: Cannot read from TLS channel: The TLS connection was non-properly terminated. warning: fd: migration to a file is deprecated. Use file: instead. warning: fd: migration to a file is deprecated. Use file: instead. ../tests/qtest/libqtest.c:205: kill_qemu() detected QEMU death from signal 11 (Segmentation fault) (core dumped) (test program exited with status code -6) TAP parsing error: Too few tests run (expected 61, got 47) ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― # Start of plain tests # Running /ppc64/migration/multifd/tcp/plain/cancel # Using machine type: pseries-9.2 # starting QEMU: exec ./qemu-system-ppc64 -qtest unix:/tmp/qtest-40766.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-40766.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -accel kvm -accel tcg -machine pseries-9.2,vsmt=8 -name source,debug-threads=on -m 256M -serial file:/tmp/migration-test-H0Z1T2/src_serial -nodefaults -machine cap-cfpc=broken,cap-sbbc=broken,cap-ibs=broken,cap-ccf-assist=off, -bios /tmp/migration-test-H0Z1T2/bootsect 2>/dev/null -accel qtest # starting QEMU: exec ./qemu-system-ppc64 -qtest unix:/tmp/qtest-40766.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-40766.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -accel kvm -accel tcg -machine pseries-9.2,vsmt=8 -name target,debug-threads=on -m 256M -serial file:/tmp/migration-test-H0Z1T2/dest_serial -incoming defer -nodefaults -machine cap-cfpc=broken,cap-sbbc=broken,cap-ibs=broken,cap-ccf-assist=off, -bios /tmp/migration-test-H0Z1T2/bootsect 2>/dev/null -accel qtest ----------------------------------- stderr ----------------------------------- qemu-system-ppc64: Cannot read from TLS channel: The TLS connection was non-properly terminated. warning: fd: migration to a file is deprecated. Use file: instead. warning: fd: migration to a file is deprecated. Use file: instead. ../tests/qtest/libqtest.c:205: kill_qemu() detected QEMU death from signal 11 (Segmentation fault) (core dumped) (test program exited with status code -6) =================================8<================================ So.. it's the same test (multifd/tcp/plain/cancel) that is failing on different host / arch being tested. What is more weird is the two failures are different, the 2nd failure throw out a TLS error even though the test doesn't yet have tls involved. Fabiano, is this the issue you're looking at? Peter, do you think it'll be helpful if we temporarily mark this test as "slow" too so it's not run in CI (so we still run it ourselves when prepare migration PR, with the hope that it can reproduce)? Thanks, -- Peter Xu
