---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <mathieu.desnoy...@efficios.com> wrote:
> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehaby...@gmail.com> > wrote: >> Yes, I tried with only one of those contexts and I still ran into the same >> problem. > What is the setting returned by > cat /proc/sys/kernel/perf_event_paranoid > on your system ? And do you run your test program as root or normal user ? > Please CC the mailing list on your reply. I will also need to know what glibc version you have on your system. Thanks, Mathieu > Thanks, > Mathieu >> Shehab Y. Elsayed, MSc. >> PhD Student >> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >> University of Toronto >> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >> shehaby...@gmail.com ] >> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [ >> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > >> wrote: >>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [ >>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>> Hi Mathieu, >>>> Thank you very much for your reply. >>>> I manually built lttng-ust from source (commit #: >>>> 8a208943e21700211beee3ea64180a5a534c7d2a). >>>> This is how I set up the tracing session: >>>> 1- lttng create lu_ncb_8_native -o {path} >>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req >>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq >>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock >>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock >>>> 3- lttng add-context -u -t procname >>>> lttng add-context -u -t vpid >>>> lttng add-context -u -t pthread_id >>>> lttng add-context -u -t vtid >>>> lttng add-context -u -t ip >>>> lttng add-context -u -t perf:thread:cpu-cycles >>>> lttng add-context -u -t perf:thread:cycles >>>> lttng add-context -u -t perf:thread:instructions >>>> 4- lttng start >>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 >>>> -n8096 >>>> -b32 >>>> 6- lttng stop >>>> 7- lttng destroy >>> Can you reproduce if you remove the following contexts ? >>> lttng add-context -u -t perf:thread:cpu-cycles >>> lttng add-context -u -t perf:thread:cycles >>> lttng add-context -u -t perf:thread:instructions >>> And if you only keep a single of those contexts ? >>> Thanks, >>> Mathieu >>>> Shehab Y. Elsayed, MSc. >>>> PhD Student >>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>> University of Toronto >>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# >>>> | >>>> shehaby...@gmail.com ] >>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [ >>>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > >>>> wrote: >>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [ >>>>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>>>> Hello All, >>>>>> I am trying to instrument a pthread application using the provided >>>>>> pthread >>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( >>>>>> fasttop >>>>>> )" error. >>>>> Please provide more information about the version of lttng-ust you are >>>>> using, >>>>> and how you setup >>>>> your tracing session. >>>>> Thanks, >>>>> Mathieu >>>>>> Here is a description of what I have tried and noticed: >>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works >>>>>> as >>>>>> expected. >>>>>> 2- From my experiments, the problem happens (more frequently at least) >>>>>> when >>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and >>>>>> instructions). >>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after >>>>>> setting LD _ >>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents. >>>>>> 4- Here is the backtrace printed after exiting: >>>>>> ======= Backtrace : ========= >>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited] >>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>> _destroy_context+0x35)[0x7ffff7471575] >>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>> _session_destroy+0x21c)[0x7ffff747363c] >>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906] >>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>> +0x9f)[0x7ffff746dccf] >>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>> +0x9f)[0x7ffff746dccf] >>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>> +0x9f)[0x7ffff746dccf] >>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi >>>>>> _exit+0x68)[0x7ffff746ead8] >>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3] >>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust >>>>>> _exit+0x67)[0x7ffff745ed57] >>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a] >>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49] >>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95] >>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51] >>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35] >>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44] >>>>>> 5- Also, this is a backtrace I obtained from gdb : >>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6 >>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6 >>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6 >>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6 >>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field ( >>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418 >>>>>> #5 0x00007ffff767a575 in lttng _destroy_context ( >>>>>> ctx =0x7ffff0011090) at lttng -context.c:278 >>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap ( >>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172 >>>>>> #7 lttng _session_destroy (session=0x7ffff0000900) >>>>>> at lttng -events.c:247 >>>>>> #8 0x00007ffff7676906 in lttng _release_session ( >>>>>> objd =<optimized out>) at lttng - ust - abi .c:601 >>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1, >>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2, >>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18, >>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216 >>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy () >>>>>> at lttng - ust - abi .c:235 >>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002 >>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1) >>>>>> at lttng - ust -comm.c:1807 >>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit () >>>>>> at lttng - ust -comm.c:1874 >>>>>> #16 0x00007ffff7dec85a in _ dl _ fini () >>>>>> from /lib64/ ld - linux -x86-64.so.2 >>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers () >>>>>> from /lib64/ libc .so.6 >>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6 >>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>, >>>>>> argv =<optimized out>) at lu .c:368 >>>>>> Any ideas, why this happens and how to fix it? >>>>>> Thanks, >>>>>> Shehab >>>>>> _______________________________________________ >>>>>> lttng-dev mailing list >>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] >>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | >>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] >>>>> -- >>>>> Mathieu Desnoyers >>>>> EfficiOS Inc. >>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>> -- >>> Mathieu Desnoyers >>> EfficiOS Inc. >>> [ http://www.efficios.com/ | http://www.efficios.com ] > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
_______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
