----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehaby...@gmail.com> wrote:
> Yes, I tried with only one of those contexts and I still ran into the same > problem. What is the setting returned by cat /proc/sys/kernel/perf_event_paranoid on your system ? And do you run your test program as root or normal user ? Please CC the mailing list on your reply. Thanks, Mathieu > Shehab Y. Elsayed, MSc. > PhD Student > The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering > University of Toronto > E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | > shehaby...@gmail.com ] > On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [ > mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > > wrote: >> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [ >> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>> Hi Mathieu, >>> Thank you very much for your reply. >>> I manually built lttng-ust from source (commit #: >>> 8a208943e21700211beee3ea64180a5a534c7d2a). >>> This is how I set up the tracing session: >>> 1- lttng create lu_ncb_8_native -o {path} >>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req >>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq >>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock >>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock >>> 3- lttng add-context -u -t procname >>> lttng add-context -u -t vpid >>> lttng add-context -u -t pthread_id >>> lttng add-context -u -t vtid >>> lttng add-context -u -t ip >>> lttng add-context -u -t perf:thread:cpu-cycles >>> lttng add-context -u -t perf:thread:cycles >>> lttng add-context -u -t perf:thread:instructions >>> 4- lttng start >>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 >>> -n8096 >>> -b32 >>> 6- lttng stop >>> 7- lttng destroy >> Can you reproduce if you remove the following contexts ? >> lttng add-context -u -t perf:thread:cpu-cycles >> lttng add-context -u -t perf:thread:cycles >> lttng add-context -u -t perf:thread:instructions >> And if you only keep a single of those contexts ? >> Thanks, >> Mathieu >>> Shehab Y. Elsayed, MSc. >>> PhD Student >>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>> University of Toronto >>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>> shehaby...@gmail.com ] >>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [ >>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > >>> wrote: >>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [ >>>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>>> Hello All, >>>>> I am trying to instrument a pthread application using the provided pthread >>>>> wrapper, but I sometimes run into a "Double free or corruption error ( >>>>> fasttop >>>>> )" error. >>>> Please provide more information about the version of lttng-ust you are >>>> using, >>>> and how you setup >>>> your tracing session. >>>> Thanks, >>>> Mathieu >>>>> Here is a description of what I have tried and noticed: >>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works >>>>> as >>>>> expected. >>>>> 2- From my experiments, the problem happens (more frequently at least) >>>>> when >>>>> adding performance counter contexts (I tried cycles, cpu _cycles and >>>>> instructions). >>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting >>>>> LD _ >>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents. >>>>> 4- Here is the backtrace printed after exiting: >>>>> ======= Backtrace : ========= >>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited] >>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>> _destroy_context+0x35)[0x7ffff7471575] >>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>> _session_destroy+0x21c)[0x7ffff747363c] >>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906] >>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>> +0x9f)[0x7ffff746dccf] >>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>> +0x9f)[0x7ffff746dccf] >>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>> +0x9f)[0x7ffff746dccf] >>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi >>>>> _exit+0x68)[0x7ffff746ead8] >>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3] >>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust >>>>> _exit+0x67)[0x7ffff745ed57] >>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a] >>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49] >>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95] >>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51] >>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35] >>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44] >>>>> 5- Also, this is a backtrace I obtained from gdb : >>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6 >>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6 >>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6 >>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6 >>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field ( >>>>> field=<optimized out>) at lttng -context- perf -counters.c:418 >>>>> #5 0x00007ffff767a575 in lttng _destroy_context ( >>>>> ctx =0x7ffff0011090) at lttng -context.c:278 >>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap ( >>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172 >>>>> #7 lttng _session_destroy (session=0x7ffff0000900) >>>>> at lttng -events.c:247 >>>>> #8 0x00007ffff7676906 in lttng _release_session ( >>>>> objd =<optimized out>) at lttng - ust - abi .c:601 >>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1, >>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2, >>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18, >>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216 >>>>> #12 0x00007ffff7677ad8 in objd _table_destroy () >>>>> at lttng - ust - abi .c:235 >>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002 >>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1) >>>>> at lttng - ust -comm.c:1807 >>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit () >>>>> at lttng - ust -comm.c:1874 >>>>> #16 0x00007ffff7dec85a in _ dl _ fini () >>>>> from /lib64/ ld - linux -x86-64.so.2 >>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers () >>>>> from /lib64/ libc .so.6 >>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6 >>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>, >>>>> argv =<optimized out>) at lu .c:368 >>>>> Any ideas, why this happens and how to fix it? >>>>> Thanks, >>>>> Shehab >>>>> _______________________________________________ >>>>> lttng-dev mailing list >>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] >>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | >>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] >>>> -- >>>> Mathieu Desnoyers >>>> EfficiOS Inc. >>>> [ http://www.efficios.com/ | http://www.efficios.com ] >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> [ http://www.efficios.com/ | http://www.efficios.com ] -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
_______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
