----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <shehaby...@gmail.com> wrote:
> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1 > I run the program as a normal user > The glibc version I get by running "ldd --version" is 2.17 Can you reproduce the issue after you do this as root ? echo "-1" > /proc/sys/kernel/perf_event_paranoid Based on this documentation of the Linux kernel: Documentation/sysctl/kernel.txt: perf_event_paranoid: Controls use of the performance events system by unprivileged users (without CAP_SYS_ADMIN). The default value is 2. -1: Allow use of (almost) all events by all users Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN Disallow raw tracepoint access by users without CAP_SYS_ADMIN >=1: Disallow CPU event access by users without CAP_SYS_ADMIN >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN Thanks, Mathieu > Shehab Y. Elsayed, MSc. > PhD Student > The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering > University of Toronto > E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | > shehaby...@gmail.com ] > On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [ > mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > > wrote: >> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [ >> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > >> wrote: >>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [ >>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>> Yes, I tried with only one of those contexts and I still ran into the same >>>> problem. >>> What is the setting returned by >>> cat /proc/sys/kernel/perf_event_paranoid >>> on your system ? And do you run your test program as root or normal user ? >>> Please CC the mailing list on your reply. >> I will also need to know what glibc version you have on your system. >> Thanks, >> Mathieu >>> Thanks, >>> Mathieu >>>> Shehab Y. Elsayed, MSc. >>>> PhD Student >>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>> University of Toronto >>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# >>>> | >>>> shehaby...@gmail.com ] >>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [ >>>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > >>>> wrote: >>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [ >>>>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>>>> Hi Mathieu, >>>>>> Thank you very much for your reply. >>>>>> I manually built lttng-ust from source (commit #: >>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a). >>>>>> This is how I set up the tracing session: >>>>>> 1- lttng create lu_ncb_8_native -o {path} >>>>>> 2- lttng enable-event --userspace >>>>>> lttng_ust_pthread:pthread_mutex_lock_req >>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq >>>>>> lttng enable-event --userspace >>>>>> lttng_ust_pthread:pthread_mutex_lock_trylock >>>>>> lttng enable-event --userspace >>>>>> lttng_ust_pthread:pthread_mutex_lock_unlock >>>>>> 3- lttng add-context -u -t procname >>>>>> lttng add-context -u -t vpid >>>>>> lttng add-context -u -t pthread_id >>>>>> lttng add-context -u -t vtid >>>>>> lttng add-context -u -t ip >>>>>> lttng add-context -u -t perf:thread:cpu-cycles >>>>>> lttng add-context -u -t perf:thread:cycles >>>>>> lttng add-context -u -t perf:thread:instructions >>>>>> 4- lttng start >>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb >>>>>> -p8 -n8096 >>>>>> -b32 >>>>>> 6- lttng stop >>>>>> 7- lttng destroy >>>>> Can you reproduce if you remove the following contexts ? >>>>> lttng add-context -u -t perf:thread:cpu-cycles >>>>> lttng add-context -u -t perf:thread:cycles >>>>> lttng add-context -u -t perf:thread:instructions >>>>> And if you only keep a single of those contexts ? >>>>> Thanks, >>>>> Mathieu >>>>>> Shehab Y. Elsayed, MSc. >>>>>> PhD Student >>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>>>> University of Toronto >>>>>> E-mail: [ >>>>>> https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>>>>> shehaby...@gmail.com ] >>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [ >>>>>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] >>>>>> > >>>>>> wrote: >>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [ >>>>>>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>>>>>> Hello All, >>>>>>>> I am trying to instrument a pthread application using the provided >>>>>>>> pthread >>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( >>>>>>>> fasttop >>>>>>>> )" error. >>>>>>> Please provide more information about the version of lttng-ust you are >>>>>>> using, >>>>>>> and how you setup >>>>>>> your tracing session. >>>>>>> Thanks, >>>>>>> Mathieu >>>>>>>> Here is a description of what I have tried and noticed: >>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes >>>>>>>> works as >>>>>>>> expected. >>>>>>>> 2- From my experiments, the problem happens (more frequently at least) >>>>>>>> when >>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and >>>>>>>> instructions). >>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after >>>>>>>> setting LD _ >>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents. >>>>>>>> 4- Here is the backtrace printed after exiting: >>>>>>>> ======= Backtrace : ========= >>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>>>> _destroy_context+0x35)[0x7ffff7471575] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>>>> _session_destroy+0x21c)[0x7ffff747363c] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi >>>>>>>> _exit+0x68)[0x7ffff746ead8] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3] >>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust >>>>>>>> _exit+0x67)[0x7ffff745ed57] >>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a] >>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49] >>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95] >>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51] >>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35] >>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44] >>>>>>>> 5- Also, this is a backtrace I obtained from gdb : >>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6 >>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6 >>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6 >>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6 >>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field ( >>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418 >>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context ( >>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278 >>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap ( >>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172 >>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900) >>>>>>>> at lttng -events.c:247 >>>>>>>> #8 0x00007ffff7676906 in lttng _release_session ( >>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601 >>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1, >>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2, >>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18, >>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216 >>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy () >>>>>>>> at lttng - ust - abi .c:235 >>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002 >>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1) >>>>>>>> at lttng - ust -comm.c:1807 >>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit () >>>>>>>> at lttng - ust -comm.c:1874 >>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini () >>>>>>>> from /lib64/ ld - linux -x86-64.so.2 >>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers () >>>>>>>> from /lib64/ libc .so.6 >>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6 >>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>, >>>>>>>> argv =<optimized out>) at lu .c:368 >>>>>>>> Any ideas, why this happens and how to fix it? >>>>>>>> Thanks, >>>>>>>> Shehab >>>>>>>> _______________________________________________ >>>>>>>> lttng-dev mailing list >>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] >>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | >>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] >>>>>>> -- >>>>>>> Mathieu Desnoyers >>>>>>> EfficiOS Inc. >>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>>> -- >>>>> Mathieu Desnoyers >>>>> EfficiOS Inc. >>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>> -- >>> Mathieu Desnoyers >>> EfficiOS Inc. >>> [ http://www.efficios.com/ | http://www.efficios.com ] >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> [ http://www.efficios.com/ | http://www.efficios.com ] -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
_______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
