On 06/11/2016 18:04, Samuel Thibault wrote:
Brian, could you run it with
export MALLOC_CHECK_=2
and also this could be useful:
export MALLOC_PERTURB_=1234
Also, to rule out the double-free scenario, and try to catch a buffer
overflow coming from the socket structure itself, I have attached a
patch which adds some debugging.
Thanks. I've added the patch, and re-run the stress test.
(Aside: since last post I've replaced the Mac Mini with Intel NUCi6KYK
so it's not *exactly* the same environment, although both machines are
quad-core i7)
Unfortunately it doesn't crash every time. Here are the first two
crashes I've managed to obtain, and they don't seem to by anything to do
with tcp_close, but I have pasted them below. I have kept the coredumps
if there's anything more useful I can extract from them.
The full command line is something like this (taken from a later run):
/usr/local/bin/qemu-system-x86_64 -m 4G -machine type=pc,accel=kvm
-netdev user,id=user.0,hostfwd=tcp::2305-:22 -device
virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0
-device virtio-net,netdev=user.0 -name vtp-nmm-201611070837.qcow2 -drive
if=none,file=output-qemu-vtp-nmm/vtp-nmm-201611070837.qcow2,id=drive0,cache=writeback,discard=unmap,format=qcow2
-boot c -vnc [::]:24
The following crashes occurred when running with a single vcpu. Normally
I have been running with -smp 8,sockets=1,cores=4,threads=2 as it seems
to crash less with those settings; however I'm trying it again like that
in a loop to see if I can get a crash.
Regards,
Brian.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/qemu-system-x86_64 -m 4G -name
vtp-nmm-201611062024.qcow2 -machi'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f366c4ce428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f366deeea80 (LWP 9030))]
(gdb) bt
#0 0x00007f366c4ce428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f366c4d002a in __GI_abort () at abort.c:89
#2 0x00007f366c4c6bd7 in __assert_fail_base (fmt=<optimised out>,
assertion=assertion@entry=0x560907875cd5 "mr != NULL",
file=file@entry=0x560907857884 "/home/nsrc/qemu-2.7.0/exec.c",
line=line@entry=2967,
function=function@entry=0x560907857f00 <__PRETTY_FUNCTION__.42881>
"address_space_unmap")
at assert.c:92
#3 0x00007f366c4c6c82 in __GI___assert_fail (
assertion=assertion@entry=0x560907875cd5 "mr != NULL",
file=file@entry=0x560907857884 "/home/nsrc/qemu-2.7.0/exec.c",
line=line@entry=2967,
function=function@entry=0x560907857f00 <__PRETTY_FUNCTION__.42881>
"address_space_unmap")
at assert.c:101
#4 0x000056090749dffe in address_space_unmap (as=<optimised out>,
buffer=<optimised out>,
len=<optimised out>, is_write=1, access_len=8192) at
/home/nsrc/qemu-2.7.0/exec.c:2967
#5 0x00005609075af586 in dma_memory_unmap (access_len=<optimised out>,
dir=<optimised out>,
len=<optimised out>, buffer=<optimised out>, as=<optimised out>)
at /home/nsrc/qemu-2.7.0/include/sysemu/dma.h:144
#6 dma_blk_unmap (dbs=dbs@entry=0x560909ceca90) at
/home/nsrc/qemu-2.7.0/dma-helpers.c:102
#7 0x00005609075af766 in dma_complete (ret=0, dbs=0x560909ceca90)
at /home/nsrc/qemu-2.7.0/dma-helpers.c:113
#8 dma_blk_cb (opaque=0x560909ceca90, ret=0) at
/home/nsrc/qemu-2.7.0/dma-helpers.c:137
#9 0x000056090775d25a in blk_aio_complete (acb=0x56090909aba0)
at /home/nsrc/qemu-2.7.0/block/block-backend.c:923
#10 0x00005609077ccaea in coroutine_trampoline (i0=<optimised out>,
i1=<optimised out>)
at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78
#11 0x00007f366c4e35d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x00007fffd8d31f20 in ?? ()
#13 0x2d2d2d2d2d2d2d2d in ?? ()
#14 0x00000000000000d0 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/qemu-system-x86_64 -boot c -vnc
[::]:78 -name vtp-nmm-2016110621'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fc226c98428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fc2286b8a80 (LWP 10267))]
(gdb) bt
#0 0x00007fc226c98428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fc226c9a02a in __GI_abort () at abort.c:89
#2 0x00007fc226c90bd7 in __assert_fail_base (fmt=<optimised out>,
assertion=assertion@entry=0x561867557cd5 "mr != NULL",
file=file@entry=0x561867539884 "/home/nsrc/qemu-2.7.0/exec.c",
line=line@entry=2967,
function=function@entry=0x561867539f00 <__PRETTY_FUNCTION__.42881>
"address_space_unmap")
at assert.c:92
#3 0x00007fc226c90c82 in __GI___assert_fail (
assertion=assertion@entry=0x561867557cd5 "mr != NULL",
file=file@entry=0x561867539884 "/home/nsrc/qemu-2.7.0/exec.c",
line=line@entry=2967,
function=function@entry=0x561867539f00 <__PRETTY_FUNCTION__.42881>
"address_space_unmap")
at assert.c:101
#4 0x000056186717fffe in address_space_unmap (as=<optimised out>,
buffer=<optimised out>,
len=<optimised out>, is_write=1, access_len=4096) at
/home/nsrc/qemu-2.7.0/exec.c:2967
#5 0x0000561867202beb in virtqueue_unmap_sg
(elem=elem@entry=0x5618694fc610, len=len@entry=32876,
vq=0x5618695a8500) at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:254
#6 0x0000561867203422 in virtqueue_fill (vq=vq@entry=0x5618695a8500,
elem=elem@entry=0x5618694fc610, len=32876, idx=idx@entry=0)
at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:282
#7 0x00005618672035db in virtqueue_push (vq=vq@entry=0x5618695a8500,
elem=elem@entry=0x5618694fc610, len=<optimised out>)
at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:308
#8 0x00005618671f0885 in virtio_scsi_complete_req (req=0x5618694fc610)
at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:70
#9 0x00005618671f09e6 in virtio_scsi_complete_cmd_req (req=0x5618694fc610)
at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:443
#10 virtio_scsi_command_complete (r=<optimised out>, status=0, resid=0)
at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:470
#11 0x0000561867365c98 in scsi_req_complete (req=0x561868a72e40,
status=<optimised out>)
at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-bus.c:1775
#12 0x0000561867360210 in scsi_dma_complete_noio (r=0x561868a72e40,
ret=<optimised out>)
at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-disk.c:278
---Type <return> to continue, or q <return> to quit---
#13 0x0000561867291779 in dma_complete (ret=0, dbs=0x561868371050)
at /home/nsrc/qemu-2.7.0/dma-helpers.c:115
#14 dma_blk_cb (opaque=0x561868371050, ret=0) at
/home/nsrc/qemu-2.7.0/dma-helpers.c:137
#15 0x000056186743f25a in blk_aio_complete (acb=0x561869661b90)
at /home/nsrc/qemu-2.7.0/block/block-backend.c:923
#16 0x00005618674aeaea in coroutine_trampoline (i0=<optimised out>,
i1=<optimised out>)
at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78
#17 0x00007fc226cad5d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x00007ffd10673d70 in ?? ()
#19 0x2d2d2d2d2d2d2d2d in ?? ()
#20 0x00000000000000d4 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)
