On Wed, Jul 10, 2024 at 9:53 PM Brian Hutchinson <b.hutch...@gmail.com> wrote:
> Hi, > > I'm getting a weird segfault when I compile in a lttng-ust tracepoint into > my application. > > I'm able to run the lttng-ust hello_world example just fine on my platform > which one would think should prove everything out, but when I use the exact > same hello world trace provider and link it into my multi-threadded app and > simply call: > > lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, > "hi there!"); > > I get a segfault. > > Background. I'm on aarch64 platform (imx8mm-lpddr4-evk) with sdk (gcc > 11.4 glibc 2.35) and rootfs generated from Yocto Kirkstone 4.0.18. Built > lttng-tools (2.13.9) and lttng-ust (2.13.6) from yocto recipe. Built > out-of-tree kernel modules for lttng-modules (2.13.9) and I'm on kernel > 6.1.38. > > If I do a backtrace with gdb on the core this is what I see: > > GNU gdb (GDB) 11.2 > Copyright (C) 2022 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "aarch64-poky-linux". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <https://www.gnu.org/software/gdb/bugs/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from ./my_app... > > warning: core file may not match specified executable file. > [New LWP 757] > [New LWP 759] > [New LWP 758] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/libthread_db.so.1". > Core was generated by `./my_app'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96 > 96 ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or > directory. > [Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))] > (gdb) bt > #0 __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96 > #1 0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from > /usr/lib/liblttng-ust-tracepoint.so.1 > #2 0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at > /opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629 > > #3 0x0000ffffb872b30c in call_init (env=<optimized out>, > argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145 > #4 __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1, > argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>, > rtld_fini=<optimized out>, stack_end=<optimized out>) at > ../csu/libc-start.c:376 > #5 0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81 > (gdb) > > If I run valgrind, this is what I see: > > ==627== Memcheck, a memory error detector > ==627== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. > ==627== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info > ==627== Command: ./my_app > ==627== > LTTng-UST: Error while registering tracepoint probe. > ==627== > ==627== Process terminating with default action of signal 6 (SIGABRT) > ==627== at 0x6772278: __pthread_kill_implementation (pthread_kill.c:44) > ==627== by 0x672DD4F: raise (raise.c:26) > ==627== by 0x671AEE3: abort (abort.c:79) > ==627== by 0x2C572B: lttng_ust__events_init__hello_world > (ust-tracepoint-event.h:1221) > ==627== by 0x2C5B97: lttng_ust_constructor_hello_world > (ust-tracepoint-event.h:1239) > ==627== by 0x671B30B: call_init (libc-start.c:145) > ==627== by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376) > ==627== by 0x21122F: (below main) (start.S:81) > ==627== > ==627== HEAP SUMMARY: > ==627== in use at exit: 151,712 bytes in 90 blocks > ==627== total heap usage: 1,184 allocs, 1,094 frees, 202,570 bytes > allocated > ==627== > ==627== 48 bytes in 1 blocks are still reachable in loss record 1 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x68D50CB: lttng_ust_tracepoint_module_register (in > /usr/lib/liblttng-ust-tracepoint.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 48 bytes in 1 blocks are still reachable in loss record 2 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x61E8143: lttng_ust_probe_register (in > /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x61FD49B: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 48 bytes in 1 blocks are still reachable in loss record 3 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x61E8143: lttng_ust_probe_register (in > /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x61DFF4F: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 48 bytes in 1 blocks are still reachable in loss record 4 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x61E8143: lttng_ust_probe_register (in > /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x61E026F: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 48 bytes in 1 blocks are still reachable in loss record 5 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x61E8143: lttng_ust_probe_register (in > /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x61E058F: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 48 bytes in 1 blocks are still reachable in loss record 6 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x68D50CB: lttng_ust_tracepoint_module_register (in > /usr/lib/liblttng-ust-tracepoint.so.1.0.0) > ==627== by 0x2C5A17: lttng_ust__tracepoints__ptrs_init > (tracepoint.h:629) > ==627== by 0x671B30B: call_init (libc-start.c:145) > ==627== by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376) > ==627== by 0x21122F: (below main) (start.S:81) > ==627== > ==627== 128 bytes in 1 blocks are still reachable in loss record 7 of 17 > ==627== at 0x618551C: malloc (vg_replace_malloc.c:381) > ==627== by 0x68A388B: ??? (in /usr/lib/liblttng-ust-common.so.1.0.0) > ==627== by 0x68A2FBF: lttng_ust_common_ctor (in > /usr/lib/liblttng-ust-common.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 200 bytes in 1 blocks are still reachable in loss record 8 of 17 > ==627== at 0x618551C: malloc (vg_replace_malloc.c:381) > ==627== by 0x5929E37: malloc (rtld-malloc.h:56) > ==627== by 0x5929E37: add_to_global_resize (dl-open.c:152) > ==627== by 0x592AA8F: dl_open_worker_begin (dl-open.c:716) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x5929C77: dl_open_worker (dl-open.c:782) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x592A07B: _dl_open (dl-open.c:884) > ==627== by 0x676C467: dlopen_doit (dlopen.c:56) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227) > ==627== by 0x676BF4F: _dlerror_run (dlerror.c:138) > ==627== by 0x676C52B: dlopen_implementation (dlopen.c:71) > ==627== by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81) > ==627== > ==627== 240 bytes in 2 blocks are still reachable in loss record 9 of 17 > ==627== at 0x618551C: malloc (vg_replace_malloc.c:381) > ==627== by 0x59214C7: malloc (rtld-malloc.h:56) > ==627== by 0x59214C7: _dl_map_object_deps (dl-deps.c:479) > ==627== by 0x592A50B: dl_open_worker_begin (dl-open.c:592) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x5929C77: dl_open_worker (dl-open.c:782) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x592A07B: _dl_open (dl-open.c:884) > ==627== by 0x676C467: dlopen_doit (dlopen.c:56) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227) > ==627== by 0x676BF4F: _dlerror_run (dlerror.c:138) > ==627== by 0x676C52B: dlopen_implementation (dlopen.c:71) > ==627== by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81) > ==627== > ==627== 292 bytes in 15 blocks are possibly lost in loss record 10 of 17 > ==627== at 0x618551C: malloc (vg_replace_malloc.c:381) > ==627== by 0x6783993: strdup (strdup.c:42) > ==627== by 0x61FD92F: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74) > ==627== by 0x61FE653: lttng_ust_dl_update (in > /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 336 bytes in 1 blocks are possibly lost in loss record 11 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x592ECB7: calloc (rtld-malloc.h:44) > ==627== by 0x592ECB7: allocate_dtv (dl-tls.c:376) > ==627== by 0x592F723: _dl_allocate_tls (dl-tls.c:635) > ==627== by 0x677114F: allocate_stack (allocatestack.c:428) > ==627== by 0x677114F: pthread_create@@GLIBC_2.34 > (pthread_create.c:647) > ==627== by 0x61DEF03: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 336 bytes in 1 blocks are possibly lost in loss record 12 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x592ECB7: calloc (rtld-malloc.h:44) > ==627== by 0x592ECB7: allocate_dtv (dl-tls.c:376) > ==627== by 0x592F723: _dl_allocate_tls (dl-tls.c:635) > ==627== by 0x677114F: allocate_stack (allocatestack.c:428) > ==627== by 0x677114F: pthread_create@@GLIBC_2.34 > (pthread_create.c:647) > ==627== by 0x61DEEB7: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 340 bytes in 17 blocks are possibly lost in loss record 13 of 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x61FD94F: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74) > ==627== by 0x61FE653: lttng_ust_dl_update (in > /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 1,248 bytes in 26 blocks are still reachable in loss record 14 of > 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x68D529F: lttng_ust_tracepoint_module_register (in > /usr/lib/liblttng-ust-tracepoint.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 4,608 bytes in 2 blocks are possibly lost in loss record 15 of 17 > ==627== at 0x618551C: malloc (vg_replace_malloc.c:381) > ==627== by 0x5922FFB: malloc (rtld-malloc.h:56) > ==627== by 0x5922FFB: _dlfo_mappings_segment_allocate > (dl-find_object.c:217) > ==627== by 0x5922FFB: _dl_find_object_update_1 (dl-find_object.c:671) > ==627== by 0x5922FFB: _dl_find_object_update (dl-find_object.c:805) > ==627== by 0x592A803: dl_open_worker_begin (dl-open.c:735) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x5929C77: dl_open_worker (dl-open.c:782) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x592A07B: _dl_open (dl-open.c:884) > ==627== by 0x676C467: dlopen_doit (dlopen.c:56) > ==627== by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208) > ==627== by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227) > ==627== by 0x676BF4F: _dlerror_run (dlerror.c:138) > ==627== by 0x676C52B: dlopen_implementation (dlopen.c:71) > ==627== by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81) > ==627== > ==627== 70,992 bytes in 17 blocks are possibly lost in loss record 16 of > 17 > ==627== at 0x618A790: calloc (vg_replace_malloc.c:1328) > ==627== by 0x61FD91B: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74) > ==627== by 0x61FE653: lttng_ust_dl_update (in > /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== 72,704 bytes in 1 blocks are still reachable in loss record 17 of > 17 > ==627== at 0x618551C: malloc (vg_replace_malloc.c:381) > ==627== by 0x653E137: ??? (in /usr/lib/libstdc++.so.6.0.29) > ==627== by 0x5923653: call_init.part.0 (dl-init.c:74) > ==627== by 0x5923767: call_init (dl-init.c:29) > ==627== by 0x5923767: _dl_init (dl-init.c:121) > ==627== by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1) > ==627== > ==627== LEAK SUMMARY: > ==627== definitely lost: 0 bytes in 0 blocks > ==627== indirectly lost: 0 bytes in 0 blocks > ==627== possibly lost: 76,904 bytes in 53 blocks > ==627== still reachable: 74,808 bytes in 37 blocks > ==627== suppressed: 0 bytes in 0 blocks > ==627== > ==627== For lists of detected and suppressed errors, rerun with: -s > ==627== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0) > Aborted > > I'm sure I'm probably doing something wrong but I've tried static linking, > dynamic linking using LD_PRELOAD helpers like > LD_PRELOAD=liblttng-ust-pthread-wrapper.so.1:liblttng-ust-fd.so.1:liblttng-ust-fork.so.1 > and I'm out of options that I know to try. > > Looking for pointers on what else I can try and what I could be doing > wrong. > > I had this same segfault issue using Dunfell version of Yocto and earlier > versions of lttng (2.11) but never dug into it because was planning on > upgrading everything soon, so recently stepped up to newer versions of > everything to see if that solved the issue but apparently not. > > I'm not sure about this, still trying to comprehend everything, but I think my issue could be related to all this gcc TLS, heap, trampoline talk here: https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html Regards, Brian
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#63470): https://lists.yoctoproject.org/g/yocto/message/63470 Mute This Topic: https://lists.yoctoproject.org/mt/107161368/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
