Hi Sudhir, Yes Asan is pretty picky about compiler version (probably issue 1), how you link your plugins (issue 3 is ASan complaining about your private plugin librtbvpp.so redefining an already defined global symbol, issue 2 might be linked to that too). Anyway, using GCC, LD_PRELOAD and ASAN_OPTIONS are valid workarounds 😊 Regarding not detecting leaks, this is something we do not support unfortunately: only memory violations (use-after-free etc) should be detected.
Best ben > -----Original Message----- > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Sudhir CR via > lists.fd.io > Sent: mardi 15 mars 2022 06:49 > To: Sudhir CR <sud...@rtbrick.com> > Cc: Benoit Ganne (bganne) <bga...@cisco.com>; vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] VPP 2110 with AddressSanitizer enabled > > Hi Ben, > > I tried to run ASAN on vpp version 21.10 in debug mode.But It's not > working for me. > > > Issue 1: > I compiled vpp code with below build command (with clang) > sudo make rebuild VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON > After compilation starting vpp is failing with below error > sed: symbol lookup error: /home/supervisor/libvpp/build-root/install- > vpp_debug-native/vpp/lib/libvppinfra.so.1.0.1: undefined symbol: > __asan_option_detect_stack_use_after_return > > > Issue 2: > since compiling with clang is not working i tried compiling with GCC. i > used below command for compilation > > > sudo make rebuild VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON > CC=gcc-8 > > > with this vpp startup is failing with below error > ==908==ASan runtime does not come first in initial library list; you > should either link runtime to your application or manually preload it with > LD_PRELOAD. > I resolved above issue with below configuration > LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libasan.so > export LD_PRELOAD > Issue 3: > once issue 2 is resolved vpp crashed with below backtrace > ``` > Thread 1 (Thread 0x7ffff7fce400 (LWP 678)): > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 > #1 0x00007ffff75917f1 in __GI_abort () at abort.c:79 > #2 0x00007fffe99f39ce in ?? () from /usr/lib/x86_64-linux- > gnu/libasan.so.5 > #3 0x00007fffe99fc088 in ?? () from /usr/lib/x86_64-linux- > gnu/libasan.so.5 > #4 0x00007fffe99d3a39 in ?? () from /usr/lib/x86_64-linux- > gnu/libasan.so.5 > #5 0x00007fffe99e1ee9 in ?? () from /usr/lib/x86_64-linux- > gnu/libasan.so.5 > #6 0x00007ffff7de38d3 in call_init (env=0x7fffffffe308, > argv=0x7fffffffe2e8, argc=3, l=<optimized out>) at dl-init.c:72 > #7 _dl_init (main_map=main_map@entry=0x5555583cf670, argc=3, > argv=0x7fffffffe2e8, env=0x7fffffffe308) at dl-init.c:119 > #8 0x00007ffff7de839f in dl_open_worker (a=a@entry=0x7fffffffcf40) at dl- > open.c:522 > #9 0x00007ffff76b816f in __GI__dl_catch_exception > (exception=0x7fffffffcf20, operate=0x7ffff7de7f60 <dl_open_worker>, > args=0x7fffffffcf40) > at dl-error-skeleton.c:196 > #10 0x00007ffff7de796a in _dl_open (file=0x7fffffffd1b0 > "/usr/local/lib/librtbvpp.so", mode=-2147483391, > caller_dlopen=0x7ffff794b73d <cmd_open_plugin+86>, nsid=<optimized > out>, argc=3, argv=<optimized out>, env=0x7fffffffe308) at dl-open.c:605 > #11 0x00007ffff70c9f96 in dlopen_doit (a=a@entry=0x7fffffffd170) at > dlopen.c:66 > #12 0x00007ffff76b816f in __GI__dl_catch_exception > (exception=exception@entry=0x7fffffffd110, operate=0x7ffff70c9f40 > <dlopen_doit>, > args=0x7fffffffd170) at dl-error-skeleton.c:196 > #13 0x00007ffff76b81ff in __GI__dl_catch_error (objname=0x55555779df30, > errstring=0x55555779df38, mallocedp=0x55555779df28, operate=<optimized > out>, > args=<optimized out>) at dl-error-skeleton.c:215 > #14 0x00007ffff70ca745 in _dlerror_run > (operate=operate@entry=0x7ffff70c9f40 <dlopen_doit>, > args=args@entry=0x7fffffffd170) at dlerror.c:162 > #15 0x00007ffff70ca051 in __dlopen (file=file@entry=0x7fffffffd1b0 > "/usr/local/lib/librtbvpp.so", mode=mode@entry=257) at dlopen.c:87 > ``` > I resolved this issue by exporting below ASAN options > export ASAN_OPTIONS=verify_asan_link_order=0:detect_odr_violation=0 > Once issue 3 is resolved vpp is up but it's not catching/reporting any > memory leaks > (I induced one leak in our code to verify the same). > we are starting our fib process (vpp) by executing below script > supervisor@dev1_sudhir>srv2:~ $ cat fib.sh > LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libasan.so > export LD_PRELOAD > export ASAN_OPTIONS=verify_asan_link_order=0:detect_odr_violation=0 > sudo -E gdb --args bd -i /etc/rtbrick/bd/config/fibd.json > > ./fib.sh > > Can you please let me know if you find any issue in the procedure i > followed or > any pointers to solve the issue i am facing. > > Thanks and Regards, > Sudhir > > On Thu, Feb 24, 2022 at 6:39 PM Sudhir CR via lists.fd.io > <http://lists.fd.io> <sudhir=rtbrick....@lists.fd.io > <mailto:rtbrick....@lists.fd.io> > wrote: > > > Hi Ben, > Thanks for the update. > > Regards, > Sudhir > > On Thu, Feb 24, 2022 at 6:37 PM Benoit Ganne (bganne) > <bga...@cisco.com <mailto:bga...@cisco.com> > wrote: > > > Hi Sudhir, > > I am working on a few bugfixes for ASan. > Right now, I'd recommend to use ASan in debug builds only, > there are several issues that need to be fixed in release mode. > > Best > ben > > > -----Original Message----- > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of Sudhir CR > via > > lists.fd.io <http://lists.fd.io> > > Sent: jeudi 24 février 2022 14:03 > > To: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > Subject: [vpp-dev] VPP 2110 with AddressSanitizer enabled > > > > Hi Team, > > > > > > We compiled vpp code with AddressSanitizer enabled. We used > the below > > command to compile code. > > > > > > sudo make rebuild-release VPP_EXTRA_CMAKE_ARGS=- > > DVPP_ENABLE_SANITIZE_ADDR=ON > > > > > > > > we are starting vpp with below command > > > > > > sudo ./vpp -c /etc/vpp/startup.conf > > > > > > > > But vpp startup is failed with below AddressSanitizer error > > > > > > AddressSanitizer:DEADLYSIGNAL > > > ================================================================= > > ==1442028==ERROR: AddressSanitizer: SEGV on unknown address > 0x0200255e3b3e > > (pc 0x7fe59ae338f7 bp 0x7ffc128b1c40 sp 0x7ffc128b1480 T0) > > ==1442028==The signal is caused by a READ memory access. > > #0 0x7fe59ae338f6 in sanitizer_unpoison_push__ > > > /home/supervisor/development/libvpp/src/vppinfra/sanitizer.h:54:17 > > #1 0x7fe59ae338f6 in hash_memory64 > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:157:15 > > #2 0x7fe59ae338f6 in hash_memory > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:280:10 > > #3 0x7fe59ae34b4e in key_sum > > /home/supervisor/development/libvpp/src/vppinfra/hash.c > > #4 0x7fe59ae34b4e in lookup.llvm.8926505759877686271 > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:557:7 > > #5 0x7fe59aeb203f in _hash_set3 > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:848:10 > > #6 0x7fe59c4edc8e in config_one_plugin > > > /home/supervisor/development/libvpp/src/vlib/unix/plugin.c:710:3 > > #7 0x7fe59c4edc8e in vlib_plugin_config > > > /home/supervisor/development/libvpp/src/vlib/unix/plugin.c:775:12 > > #8 0x7fe59c49c3f6 in vlib_unix_main > > > /home/supervisor/development/libvpp/src/vlib/unix/main.c:764:12 > > #9 0x55a040ce2e40 in main > > > /home/supervisor/development/libvpp/src/vpp/vnet/main.c:344:14 > > #10 0x7fe59a29ebf6 in __libc_start_main /build/glibc- > S9d2JN/glibc- > > 2.27/csu/../csu/libc-start.c:310 > > #11 0x55a040c41479 in _start > > (/home/supervisor/development/libvpp/build-root/install-vpp- > > native/vpp/bin/vpp+0x41479) > > > > AddressSanitizer can not provide additional info. > > SUMMARY: AddressSanitizer: SEGV > > > /home/supervisor/development/libvpp/src/vppinfra/sanitizer.h:54:17 in > > sanitizer_unpoison_push__ > > ==1442028==ABORTING > > Aborted > > > > > > > > contents of startup.conf looks like below > > > > > > > > unix { > > nodaemon > > log /var/log/vpp/vpp.log > > full-coredump > > cli-listen localhost:5003 > > runtime-dir /shm/run/vpp/ > > } > > > > api-trace { > > on > > } > > > > cpu { > > main-core 2 > > corelist-workers 3,4 > > } > > > > heapsize 600M > > > > statseg { > > size 150M > > } > > plugin_path /usr/local/lib/vpp_plugins/ > > plugins { > > plugin rtbrick_bcm_plugin.so { disable } > > plugin dpdk_plugin.so { disable } > > } > > > > > > > > Please let me know any suggestions on how to resolve this > Error. > > > > > > Thanks in Advance, > > > > Sudhir > > > > > > NOTICE TO RECIPIENT This e-mail message and any attachments > are > > confidential and may be privileged. If you received this e- > mail in error, > > any review, use, dissemination, distribution, or copying of > this e-mail is > > strictly prohibited. Please notify us immediately of the > error by return > > e-mail and please delete this message from your system. For > more > > information about Rtbrick, please visit us at > www.rtbrick.com <http://www.rtbrick.com> > > <http://www.rtbrick.com> > > > > NOTICE TO RECIPIENT This e-mail message and any attachments are > confidential and may be privileged. If you received this e-mail in error, > any review, use, dissemination, distribution, or copying of this e-mail is > strictly prohibited. Please notify us immediately of the error by return > e-mail and please delete this message from your system. For more > information about Rtbrick, please visit us at www.rtbrick.com > <http://www.rtbrick.com> > > > > > > NOTICE TO RECIPIENT This e-mail message and any attachments are > confidential and may be privileged. If you received this e-mail in error, > any review, use, dissemination, distribution, or copying of this e-mail is > strictly prohibited. Please notify us immediately of the error by return > e-mail and please delete this message from your system. For more > information about Rtbrick, please visit us at www.rtbrick.com > <http://www.rtbrick.com>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21025): https://lists.fd.io/g/vpp-dev/message/21025 Mute This Topic: https://lists.fd.io/mt/89364107/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
