I've noticed recently that intermittently 'make check' will hang on
my aarch32 test system (really an aarch64 box with an aarch32 chroot).
I think from grep that this must be the vhost-user-blk test.
Here's the process tree:
pmaydell 13126 0.0 0.0 8988 6416 ? S Jul09 0:01 make
-C build/all-a32 check V=1 GCC_COLORS= -j9
pmaydell 19632 0.0 0.0 4432 2096 ? S Jul09 0:00 \_
bash -o pipefail -c echo 'MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((
${RANDOM:-0} % 255 + 1))} QTEST_QEMU_IMG=./qemu-img
G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
QTEST_QEMU_BINARY=./qemu-system-i386
QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
tests/qtest/qos-test --tap -k' &&
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
QTEST_QEMU_IMG=./qemu-img
G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
QTEST_QEMU_BINARY=./qemu-system-i386
QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
tests/qtest/qos-test --tap -k -m quick < /dev/null |
./scripts/tap-driver.pl --test-name="qtest-i386/qos-test"
pmaydell 19634 0.0 0.0 13608 3076 ? Sl Jul09 0:02
\_ tests/qtest/qos-test --tap -k -m quick
pmaydell 20679 0.0 0.0 109076 16100 ? Sl Jul09 0:00
| \_ ./storage-daemon/qemu-storage-daemon --blockdev
driver=file,node-name=disk0,filename=qtest.X7RL2X --export
type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-19634-sock.9LJoHn,node-name=disk0,writable=on,num-queues=1
pmaydell 20681 0.0 0.2 447828 46544 ? Sl Jul09 0:00
| \_ ./qemu-system-i386 -qtest unix:/tmp/qtest-19634.sock -qtest-log
/dev/null -chardev socket,path=/tmp/qtest-19634.qmp,id=char0 -mon
chardev=char0,mode=control -display none -M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-sock.9LJoHn
-accel qtest
pmaydell 19635 0.0 0.0 10256 7176 ? S Jul09 0:00
\_ perl ./scripts/tap-driver.pl --test-name=qtest-i386/qos-test
Backtrace from tests/qtest/qos-test (not as helpful as it could
be since this is an optimized build):
(gdb) thread apply all bt
Thread 2 (Thread 0xf76ff240 (LWP 19636)):
#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
#1 0x005206de in qemu_futex_wait (val=<optimised out>, f=<optimised
out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0x5816fc <rcu_call_ready_event>) at
../../util/qemu-thread-posix.c:480
#3 0x005469c2 in call_rcu_thread (opaque=<optimised out>) at
../../util/rcu.c:258
#4 0x0051fbc2 in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#5 0xf785a614 in start_thread (arg=0xf6ce711c) at pthread_create.c:463
#6 0xf77f57ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 1 (Thread 0xf7a04010 (LWP 19634)):
#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf7861d8c in __libc_read (fd=12, buf=buf@entry=0xff9ce8e4,
nbytes=nbytes@entry=1024) at ../sysdeps/unix/sysv/linux/read.c:27
#2 0x004ebc5a in read (__nbytes=1024, __buf=0xff9ce8e4,
__fd=<optimised out>) at
/usr/include/arm-linux-gnueabihf/bits/unistd.h:44
#3 qtest_client_socket_recv_line (s=0x1a46cb8) at
../../tests/qtest/libqtest.c:494
#4 0x004ebd4e in qtest_rsp_args (s=s@entry=0x1a46cb8,
expected_args=expected_args@entry=1) at
../../tests/qtest/libqtest.c:521
#5 0x004ec1ee in qtest_query_target_endianness (s=0x1a46cb8) at
../../tests/qtest/libqtest.c:570
#6 0x004ec94a in qtest_init_without_qmp_handshake
(extra_args=<optimised out>) at ../../tests/qtest/libqtest.c:332
#7 0x004ecd7a in qtest_init (extra_args=<optimised out>) at
../../tests/qtest/libqtest.c:339
#8 0x004ded10 in qtest_start (
args=0x1a63710 "-M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
../../tests/qtest/libqtest-single.h:29
#9 restart_qemu_or_continue (
path=0x1a63710 "-M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
../../tests/qtest/qos-test.c:105
#10 run_one_test (arg=<optimised out>) at ../../tests/qtest/qos-test.c:178
#11 0xf794ee74 in ?? () from /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Backtrace from qemu-system-i386:
(gdb) thread apply all bt
Thread 4 (Thread 0xdfd0cb90 (LWP 20734)):
#0 0xf6f85206 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
#1 0xf6f93492 in __GI___sigtimedwait (set=<optimised out>,
set@entry=0xdfd0c3c4, info=info@entry=0xdfd0c324,
timeout=timeout@entry=0x0) at
../sysdeps/unix/sysv/linux/sigtimedwait.c:42
#2 0xf7073e6c in __sigwait (set=set@entry=0xdfd0c3c4,
sig=sig@entry=0xdfd0c3c0) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#3 0x00c999a6 in dummy_cpu_thread_fn (arg=0x24d1168) at
../../accel/dummy-cpus.c:46
#4 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#5 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
#6 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
Thread 3 (Thread 0xe07fcb90 (LWP 20710)):
#0 0xf6f85204 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf6fff948 in __GI___poll (fds=0x2409b90, nfds=3, timeout=-1) at
../sysdeps/unix/sysv/linux/poll.c:29
#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
Thread 2 (Thread 0xf1201b90 (LWP 20703)):
#0 0xf7074f04 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf7072ad2 in __lll_lock_wait (futex=futex@entry=0x14f2e68
<qemu_global_mutex>, private=0) at lowlevellock.c:43
#2 0xf706d3e2 in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x14f2e68 <qemu_global_mutex>) at
pthread_mutex_lock.c:78
#3 0x00e18c9c in qemu_mutex_lock_impl (mutex=0x14f2e68
<qemu_global_mutex>, file=0x1355bd4 "../../util/rcu.c", line=266)
at ../../util/qemu-thread-posix.c:79
#4 0x00c1403a in qemu_mutex_lock_iothread_impl (file=0x1355bd4
"../../util/rcu.c", line=line@entry=266) at ../../softmmu/cpus.c:493
#5 0x00e15ad4 in call_rcu_thread (opaque=<optimised out>) at
../../util/rcu.c:266
#6 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#7 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
#8 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
Thread 1 (Thread 0xf12f9010 (LWP 20681)):
#0 0xf6f85204 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf6fff948 in __GI___poll (fds=0x3248040, nfds=2, timeout=-1) at
../sysdeps/unix/sysv/linux/poll.c:29
#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace of qemu-storage-daemon:
(gdb) thread apply all bt
Thread 2 (Thread 0xf21ffdd0 (LWP 20722)):
#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
#1 0x009a9986 in qemu_futex_wait (val=<optimised out>, f=<optimised
out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0xa64958 <rcu_call_ready_event>) at
../../util/qemu-thread-posix.c:480
#3 0x009a5d66 in call_rcu_thread (opaque=<optimised out>) at
../../util/rcu.c:258
#4 0x009a8e6a in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#5 0xf79dd614 in start_thread (arg=0xa310c11f) at pthread_create.c:463
#6 0xf79787ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 1 (Thread 0xf22f7010 (LWP 20679)):
#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf7971a04 in __GI_ppoll (fds=0x117c578, nfds=5,
timeout=<optimised out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:39
#2 0x009a68fc in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimised
out>, __fds=<optimised out>)
at /usr/include/arm-linux-gnueabihf/bits/poll2.h:77
#3 qemu_poll_ns (fds=<optimised out>, nfds=<optimised out>,
timeout=<optimised out>) at ../../util/qemu-timer.c:336
#4 0x009c0222 in os_host_main_loop_wait (timeout=-1) at
../../util/main-loop.c:250
#5 main_loop_wait (nonblocking=<optimised out>) at ../../util/main-loop.c:531
#6 0x008ee92a in main (argc=5, argv=0xffdce5c4) at
../../storage-daemon/qemu-storage-daemon.c:345
Any ideas ?
thanks
-- PMM
