Hi, it seems that using virtio-scsi did the trick, But now the VMs are pausing without an coredump, so the underlying Problem (no storage Error) is not fixed, As I am using Snapshots (and so the disks have to grow very fast) I try if tuning "volume_utilization_percent" and "volume_utilization_chunk_mb" will help (https://access.redhat.com/solutions/130843)
Thx Christian -----Ursprüngliche Nachricht----- Von: Dr. David Alan Gilbert [mailto:dgilb...@redhat.com] Gesendet: Dienstag, 17. November 2015 15:42 An: Grundmann, Christian <christian.grundm...@fabasoft.com> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefa...@redhat.com Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc * Grundmann, Christian (christian.grundm...@fabasoft.com) wrote: > And here another one Oh this is a bit of a different one, from query-blockstats, although again if the heap's corrupted it might have just been the first guy to trip over the corrupt part afterwards. Dave > Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)): > #0 0x00007f5cff7e2e7d in > tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from > /lib64/libtcmalloc.so.4 No symbol table info available. > #1 0x00007f5cff7e312a in > tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from > /lib64/libtcmalloc.so.4 No symbol table info available. > #2 0x00007f5cff7e31dd in > tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from > /lib64/libtcmalloc.so.4 No symbol table info available. > #3 0x00007f5cff7e6235 in > tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () > from /lib64/libtcmalloc.so.4 No symbol table info available. > #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 No > symbol table info available. > #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575 > ptr = 0x1 > #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, > key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79 > entry = 0x7f5d088df480 > #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb > "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145 > bucket = 81 > entry = <optimized out> > #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields > (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at > qapi-visit.c:1542 > err = 0x0 > #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, > obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", > errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566 > err = 0x0 > #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, > obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614 > err = 0x0 > #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, > name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at > qapi-visit.c:1644 > err = 0x0 > #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, > obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620 > err = 0x0 > #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, > name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644 > err = 0x0 > #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, > obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", > errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665 > native_i = <optimized out> > err = 0x0 > i = 0x7f5d07c67a50 > prev = 0x7ffddb417d40 > #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats > (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at > qmp-marshal.c:182 > local_err = 0x0 > mo = 0x7f5d061fdea0 > md = <optimized out> > v = <optimized out> > #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized > out>, ret=0x7ffddb417e10) at qmp-marshal.c:225 > local_err = 0x0 > args = <optimized out> > retval = <optimized out> > mi = 0x7f5d064e2000 > md = <optimized out> > v = <optimized out> > has_query_nodes = false > query_nodes = false > #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, > params=0x7f5d075dd600, mon=0x7f5d06208320) at > /usr/src/debug/qemu-2.3.0/monitor.c:5051 > ret = <optimized out> > data = 0x0 > #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at > /usr/src/debug/qemu-2.3.0/monitor.c:5113 > err = <optimized out> > obj = <optimized out> > input = <optimized out> > args = 0x7f5d075dd600 > cmd_name = <optimized out> > mon = 0x7f5d06208320 > #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, > token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at > qobject/json-streamer.c:87 > parser = 0x7f5d061f5d68 > dict = 0x7f5d088ea800 > #22 0x00007f5d03ef191f in json_lexer_feed_char > (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, > flush=flush@entry=false) at qobject/json-lexer.c:303 > new_state = 100 > #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, > buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 > err = <optimized out> > i = <optimized out> > #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized > out>, buffer=<optimized out>, size=<optimized out>) at > qobject/json-streamer.c:110 No locals. > #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, > buf=<optimized out>, size=<optimized out>) at > /usr/src/debug/qemu-2.3.0/monitor.c:5134 > old_mon = 0x0 > #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, > buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305 > No locals. > #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, > opaque=0x7f5d0625a2e0) at qemu-char.c:2870 > chr = 0x7f5d0625a2e0 > s = 0x7f5d061aa3f0 > buf = > "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", > '\000' <repeats 18 times>, > "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 > \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 > > Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"... > len = <optimized out> > size = <optimized out> > #28 0x00007f5d020b099a in g_main_context_dispatch () from > /lib64/libglib-2.0.so.0 No symbol table info available. > #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209 > context = 0x7f5d06205140 > pfds = <optimized out> > #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 > ret = 2 > spin_counter = 0 > #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 > ret = 2 > timeout = 4294967295 > timeout_ns = <optimized out> > #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 2 > #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) > at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7f5d03f06a67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 67 > optarg = 0x7f5d06193570 "rhel6.5.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 56 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = > 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, > calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > -----Ursprüngliche Nachricht----- > Von: qemu-devel-bounces+christian.grundmann=fabasoft....@nongnu.org > [mailto:qemu-devel-bounces+christian.grundmann=fabasoft....@nongnu.org > ] Im Auftrag von Grundmann, Christian > Gesendet: Dienstag, 17. November 2015 15:12 > An: 'Dr. David Alan Gilbert' <dgilb...@redhat.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > stefa...@redhat.com > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > libtcmalloc > > Here you go > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set > pagination off" -ex "thread apply all bt full" > [New LWP 52281] > [New LWP 52288] > [New LWP 52286] > [New LWP 52291] > [New LWP 52292] > [New LWP 52287] > [New LWP 52293] > [New LWP 52290] > [New LWP 56455] > [New LWP 52289] > [New LWP 52282] > [Thread debugging using libthread_db enabled] Using host libthread_db library > "/lib64/libthread_db.so.1". > Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine > rhel6.5.0,accel=kvm,us'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 > No locals. > #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 > <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. > #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at > util/qemu-thread-posix.c:399 > value = <optimized out> > #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at > util/rcu.c:233 > tries = 0 > n = <optimized out> > node = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d47719700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, > 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, > -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570cf000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8a4000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d44f14700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, > 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, > -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): > #0 sem_timedwait () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 > No locals. > #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, > ms=ms@entry=10000) at util/qemu-thread-posix.c:254 > rc = <optimized out> > ts = {tv_sec = 1447709021, tv_nsec = 21985000} > __func__ = "qemu_sem_timedwait" > #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at > thread-pool.c:92 > req = <optimized out> > ret = <optimized out> > pool = 0x7f6d5a1b91e0 > #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6ab1dff700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, > 5884348200482620104, 0, 140096227506624, 140096227505920, 26, > -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #4 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570cc000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8b8000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d44713700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, > 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, > -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c3000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8f4000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d42f10700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, > 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, > -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d5000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c878000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d45f16700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, > 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, > -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c6000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8e0000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d43711700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, > 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, > -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c9000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8cc000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d43f12700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, > 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, > -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d8000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c810000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d46717700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, > 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, > -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, > type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at > /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d2000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at > /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c890000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at > pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d45715700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, > 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, > -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = > {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol > table info available. > #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575 > ptr = 0x7f6d59a346a0 > #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol > table info available. > #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at > /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33 > req = <optimized out> > #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at > /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192 > req = <optimized out> > #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at > /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604 > s = 0x7f6d5de1ff40 > __func__ = "virtio_blk_handle_output" > mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, > 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true} > #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, > ret=62, ret@entry=1) at iohandler.c:143 > revents = 1 > #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at > main-loop.c:504 > ret = 1 > timeout = 4294967295 > timeout_ns = <optimized out> > #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 0 > #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) > at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7f6d57449a67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 69 > optarg = 0x7f6d5a14b3a0 "rhel6.5.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 179 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = > 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, > calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > > @ Do you think you're only hitting these crashes on VMs that have been paused > because of these space errors? > Will have a look on that > > Thx Christian > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilb...@redhat.com] > Gesendet: Dienstag, 17. November 2015 12:36 > An: Grundmann, Christian <christian.grundm...@fabasoft.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > stefa...@redhat.com > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > libtcmalloc > > * Grundmann, Christian (christian.grundm...@fabasoft.com) wrote: > > Hi, > > > > @ Can you please use a 'thread apply all bt full' the full gives a little > > more info. > > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set > > pagination off" -ex "thread apply all bt full" > > OK, it doesn't relaly give any more without the debuginfo package mentioned > below. > > <snip> > > > @ Also, if you've not already got it installed can you please install the > > debuginfo package for qemu, it gives a lot more information in backtraces. > > Sorry it's a ovirt-node System where I can't you yum > > Ah, although perhaps if you took the core dump, onto another machine with > matching qemu and debuginfo you should be able to get more detail. > > > @ Does this part always look the same in your backtraces? > > The most are the same, found one a little bit different : > > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > > #0 0x00007f3785d18353 in > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > > unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info > > available. > > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from > > /lib64/libtcmalloc.so.4 No symbol table info available. > > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No > > symbol table info available. > > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No > > symbol table info available. > > #4 0x00007f37885fec89 in g_slice_free1 () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table > > info available. > > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info > > available. > > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info > > available. > > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table > > info available. > > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info > > available. > > #10 0x00007f37885e299a in g_main_context_dispatch () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info > > available. > > #12 0x00007f378a1a5a4e in main () > > No symbol table info available. > > > > OK, that's a bit different but interesting.... > > > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > No nothing abnormal > > > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a > > problem? > > Ovirt tells me "no Storage space error". Which is something like the disk > > is growing to fast i think. I use Snapshots so on heavy write the disk has > > to grow a lot. > > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays > > offline. > > OK, that's interesting, because you may be hitting the following bug; > http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html > > whose fix coincidentally just got accepted today; it's related to error cases > with error=stop which you are using. > > Do you think you're only hitting these crashes on VMs that have been paused > because of these space errors? > > > disk emulation and see if the problem goes away - e.g. virtio-scsi > > would be a good one to try. > > > > Ok will try that and report > > Thanks, > > Dave > > > > > Thx Christian > > > > > > -----Ursprüngliche Nachricht----- > > Von: Dr. David Alan Gilbert [mailto:dgilb...@redhat.com] > > Gesendet: Dienstag, 17. November 2015 10:59 > > An: Grundmann, Christian <christian.grundm...@fabasoft.com> > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > > stefa...@redhat.com > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > > libtcmalloc > > > > * Grundmann, Christian (christian.grundm...@fabasoft.com) wrote: > > > Hi, > > > Dan sent me over to you, > > > please let me know if i can provide additional informations > > > > Hi Christian, > > Thanks for reporting this, > > > > > Softwareversions: > > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > > kernel-3.10.0-229.14.1.el7.x86_64 > > > gperftools-libs-2.4-7.el7.x86_64 > > > > > > Commandline: > > > /usr/libexec/qemu-kvm -name myvmname -S -machine > > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime > > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > > > type=1,manufacturer=oVirt,product=oVirt > > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-313 > > > 8- > > > 5A > > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > > -nographic -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor, > > > se rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control > > > -rtc base=2015-11-15T20:04:35,driftfix=slew -global > > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot > > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > > > -device > > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x > > > 5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > > -drive > > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b > > > 84 > > > -8 > > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baff > > > ae > > > cf > > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk > > > 0, > > > fo > > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none, > > > we rr or=stop,rerror=stop,aio=native -device > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk > > > 0, > > > id > > > =virtio-disk0,bootindex=1 -netdev > > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=p > > > ci > > > .0 > > > ,addr=0x3 -chardev > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b889 > > > 9- 5a > > > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait > > > -device > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=c > > > ha nn el0,name=com.redhat.rhevm.vdsm -chardev > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b889 > > > 9- 5a > > > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait > > > -device > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=c > > > ha > > > nn > > > el1,name=org.qemu.guest_agent.0 -device > > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg > > > timestamp=on > > > > > > Stack Trace: > > > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set > > > pagination off" -ex "thread apply all bt" > > > > Can you please use a 'thread apply all bt full' the full gives a little > > more info. > > Also, if you've not already got it installed can you please install the > > debuginfo package for qemu, it gives a lot more information in backtraces. > > > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > > #0 0x00007fa8ad2febe1 in tc_malloc () from > > > /lib64/libtcmalloc.so.4 > > > #1 0x00007fa8b186b489 in malloc_and_trace () > > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > > /lib64/libglib-2.0.so.0 > > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > > #6 0x00007fa8b197e296 in main_loop_wait () > > > #7 0x00007fa8b177da4e in main () > > > > Does this part always look the same in your backtraces? > > The segfault in tc_malloc is probably due to a heap corruption, or double > > free or similar - although it can be a bit tricky to find out what did it, > > since the corruption might have happened a bit before the place it crashed. > > > > Some other ideas: > > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > 2) Did you hit any IO errors and need to tell the VM to continue after a > > problem? > > 3) If this is pretty repeatable, then it would be interesting to try > > changing to a different > > disk emulation and see if the problem goes away - e.g. virtio-scsi > > would be a good one to try. > > > > Dave > > > > > > > > > Thx Christian > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Dan Kenigsberg [mailto:dan...@redhat.com] > > > Gesendet: Freitag, 13. November 2015 20:00 > > > An: Grundmann, Christian <christian.grundm...@fabasoft.com> > > > Cc: 'us...@ovirt.org' <us...@ovirt.org> > > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > > > Hi, > > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > > > (is there something better to use?) fort he nodes, and have > > > > random crashes of VMs The dumps are always the Same > > > > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump > > > > [Thread debugging using libthread_db enabled] Using host > > > > libthread_db library "/lib64/libthread_db.so.1". > > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine > > > > rhel6.5.0,accel=kvm,usb=o'. > > > > Program terminated with signal 11, Segmentation fault. > > > > #0 0x00007f0c559c4353 in > > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > > centos7 or 3.6 > > > > > > Due to the low-leveled-ness of the problem, I'd guess it's a > > > qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > > > > Please report the precise version of qemu,kernel,libvirt and > > > gperftools-libs to qemu-devel mailing list and the complete stack trace > > > and qemu command line, if possible. > > > > > -- > > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
