Hi Mark,
On Tuesday 13 June 2017 05:14 PM, Mark Wielaard wrote:
> I see the same on very short runs. But when doing a slightly longer run,
> even just using ls -lahR, which does some more work, then I do see user
> backtraces. They are still missing for some of the early samples though.
> It is as if there is a stack/memory address mismatch when the probe is
> "too early" in ld.so.
>
> Could you do a test run on some program that does some more work to see
> if you never get any user stack traces, or if you only not get them for
> some specific probes?
Thanks for checking. I tried a proper workload this time, but I still
don't see any userspace callchain getting unwound.
$ ./perf record --call-graph=dwarf -- zip -q -r temp.zip .
[ perf record: Woken up 2891 times to write data ]
[ perf record: Captured and wrote 723.290 MB perf.data (87934 samples) ]
With libdw:
$ LD_LIBRARY_PATH=/home/ravi/elfutils-git/usr/local/lib:\
/home/ravi/elfutils-git/usr/local/lib/elfutils/:$LD_LIBRARY_PATH\
./perf script
zip 16699 6857.354633: 37371 cycles:u:
ecedc xmon_core
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
8c4fc __hash_page_64K
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
83450 hash_preload
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7cc34 update_mmu_cache
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
330064 alloc_set_pte
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
330efc do_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
334580 __handle_mm_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
335040 handle_mm_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7bf94 do_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7bec4 do_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7be78 do_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
1a4f8 handle_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
zip 16699 6857.354663: 300677 cycles:u:
zip 16699 6857.354895: 584131 cycles:u:
zip 16699 6857.355312: 589687 cycles:u:
zip 16699 6857.355606: 560142 cycles:u:
With libunwind:
$ ./perf script
zip 16699 6857.354633: 37371 cycles:u:
ecedc xmon_core
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
8c4fc __hash_page_64K
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
83450 hash_preload
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7cc34 update_mmu_cache
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
330064 alloc_set_pte
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
330efc do_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
334580 __handle_mm_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
335040 handle_mm_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7bf94 do_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7bec4 do_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
7be78 do_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
1a4f8 handle_page_fault
(/usr/lib/debug/lib/modules/4.11.0-3.el7.ppc64le/vmlinux)
1920 _start (/usr/lib64/ld-2.17.so)
zip 16699 6857.354663: 300677 cycles:u:
fa38 _dl_new_object (/usr/lib64/ld-2.17.so)
3073 dl_main (/usr/lib64/ld-2.17.so)
2045b _dl_sysdep_start (/usr/lib64/ld-2.17.so)
1c7f _dl_start_final (/usr/lib64/ld-2.17.so)
5ce7 _dl_start (/usr/lib64/ld-2.17.so)
1937 _start (/usr/lib64/ld-2.17.so)
zip 16699 6857.354895: 584131 cycles:u:
103d0 _dl_relocate_object (/usr/lib64/ld-2.17.so)
zip 16699 6857.355312: 589687 cycles:u:
df68 do_lookup_x (/usr/lib64/ld-2.17.so)
e8d7 _dl_lookup_symbol_x (/usr/lib64/ld-2.17.so)
14bb3 _dl_fixup (/usr/lib64/ld-2.17.so)
1ef37 _dl_runtime_resolve (/usr/lib64/ld-2.17.so)
20bf7 copy_args (/usr/bin/zip)
286f main (/usr/bin/zip)
2497f generic_start_main.isra.0 (/usr/lib64/libc-2.17.so)
24b73 __libc_start_main (/usr/lib64/libc-2.17.so)
zip 16699 6857.355606: 560142 cycles:u:
84764 _IO_getc (/usr/lib64/libc-2.17.so)
c4d7 find_next_signature (/usr/bin/zip)
15c8b readzipfile (/usr/bin/zip)
3477 main (/usr/bin/zip)
2497f generic_start_main.isra.0 (/usr/lib64/libc-2.17.so)
24b73 __libc_start_main (/usr/lib64/libc-2.17.so)
Right now, I'm busy with few other things so I'm not able to dig further.
But I can look into this next week or so.
Thanks,
Ravi
