Hi Ben, Thank you very much for the response and useful information. Regards, Sudhir
On Tue, Mar 15, 2022 at 11:02 PM Benoit Ganne (bganne) <bga...@cisco.com> wrote: > Hi Sudhir, > > Yes Asan is pretty picky about compiler version (probably issue 1), how > you link your plugins (issue 3 is ASan complaining about your private > plugin librtbvpp.so redefining an already defined global symbol, issue 2 > might be linked to that too). > Anyway, using GCC, LD_PRELOAD and ASAN_OPTIONS are valid workarounds 😊 > Regarding not detecting leaks, this is something we do not support > unfortunately: only memory violations (use-after-free etc) should be > detected. > > Best > ben > > > -----Original Message----- > > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Sudhir CR > via > > lists.fd.io > > Sent: mardi 15 mars 2022 06:49 > > To: Sudhir CR <sud...@rtbrick.com> > > Cc: Benoit Ganne (bganne) <bga...@cisco.com>; vpp-dev@lists.fd.io > > Subject: Re: [vpp-dev] VPP 2110 with AddressSanitizer enabled > > > > Hi Ben, > > > > I tried to run ASAN on vpp version 21.10 in debug mode.But It's not > > working for me. > > > > > > Issue 1: > > I compiled vpp code with below build command (with clang) > > sudo make rebuild VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON > > After compilation starting vpp is failing with below error > > sed: symbol lookup error: /home/supervisor/libvpp/build-root/install- > > vpp_debug-native/vpp/lib/libvppinfra.so.1.0.1: undefined symbol: > > __asan_option_detect_stack_use_after_return > > > > > > Issue 2: > > since compiling with clang is not working i tried compiling with GCC. i > > used below command for compilation > > > > > > sudo make rebuild VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON > > CC=gcc-8 > > > > > > with this vpp startup is failing with below error > > ==908==ASan runtime does not come first in initial library list; you > > should either link runtime to your application or manually preload it > with > > LD_PRELOAD. > > I resolved above issue with below configuration > > LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libasan.so > > export LD_PRELOAD > > Issue 3: > > once issue 2 is resolved vpp crashed with below backtrace > > ``` > > Thread 1 (Thread 0x7ffff7fce400 (LWP 678)): > > #0 __GI_raise (sig=sig@entry=6) at > ../sysdeps/unix/sysv/linux/raise.c:51 > > #1 0x00007ffff75917f1 in __GI_abort () at abort.c:79 > > #2 0x00007fffe99f39ce in ?? () from /usr/lib/x86_64-linux- > > gnu/libasan.so.5 > > #3 0x00007fffe99fc088 in ?? () from /usr/lib/x86_64-linux- > > gnu/libasan.so.5 > > #4 0x00007fffe99d3a39 in ?? () from /usr/lib/x86_64-linux- > > gnu/libasan.so.5 > > #5 0x00007fffe99e1ee9 in ?? () from /usr/lib/x86_64-linux- > > gnu/libasan.so.5 > > #6 0x00007ffff7de38d3 in call_init (env=0x7fffffffe308, > > argv=0x7fffffffe2e8, argc=3, l=<optimized out>) at dl-init.c:72 > > #7 _dl_init (main_map=main_map@entry=0x5555583cf670, argc=3, > > argv=0x7fffffffe2e8, env=0x7fffffffe308) at dl-init.c:119 > > #8 0x00007ffff7de839f in dl_open_worker (a=a@entry=0x7fffffffcf40) at > dl- > > open.c:522 > > #9 0x00007ffff76b816f in __GI__dl_catch_exception > > (exception=0x7fffffffcf20, operate=0x7ffff7de7f60 <dl_open_worker>, > > args=0x7fffffffcf40) > > at dl-error-skeleton.c:196 > > #10 0x00007ffff7de796a in _dl_open (file=0x7fffffffd1b0 > > "/usr/local/lib/librtbvpp.so", mode=-2147483391, > > caller_dlopen=0x7ffff794b73d <cmd_open_plugin+86>, nsid=<optimized > > out>, argc=3, argv=<optimized out>, env=0x7fffffffe308) at dl-open.c:605 > > #11 0x00007ffff70c9f96 in dlopen_doit (a=a@entry=0x7fffffffd170) at > > dlopen.c:66 > > #12 0x00007ffff76b816f in __GI__dl_catch_exception > > (exception=exception@entry=0x7fffffffd110, operate=0x7ffff70c9f40 > > <dlopen_doit>, > > args=0x7fffffffd170) at dl-error-skeleton.c:196 > > #13 0x00007ffff76b81ff in __GI__dl_catch_error (objname=0x55555779df30, > > errstring=0x55555779df38, mallocedp=0x55555779df28, operate=<optimized > > out>, > > args=<optimized out>) at dl-error-skeleton.c:215 > > #14 0x00007ffff70ca745 in _dlerror_run > > (operate=operate@entry=0x7ffff70c9f40 <dlopen_doit>, > > args=args@entry=0x7fffffffd170) at dlerror.c:162 > > #15 0x00007ffff70ca051 in __dlopen (file=file@entry=0x7fffffffd1b0 > > "/usr/local/lib/librtbvpp.so", mode=mode@entry=257) at dlopen.c:87 > > ``` > > I resolved this issue by exporting below ASAN options > > export ASAN_OPTIONS=verify_asan_link_order=0:detect_odr_violation=0 > > Once issue 3 is resolved vpp is up but it's not catching/reporting any > > memory leaks > > (I induced one leak in our code to verify the same). > > we are starting our fib process (vpp) by executing below script > > supervisor@dev1_sudhir>srv2:~ $ cat fib.sh > > LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/8/libasan.so > > export LD_PRELOAD > > export ASAN_OPTIONS=verify_asan_link_order=0:detect_odr_violation=0 > > sudo -E gdb --args bd -i /etc/rtbrick/bd/config/fibd.json > > > > ./fib.sh > > > > Can you please let me know if you find any issue in the procedure i > > followed or > > any pointers to solve the issue i am facing. > > > > Thanks and Regards, > > Sudhir > > > > On Thu, Feb 24, 2022 at 6:39 PM Sudhir CR via lists.fd.io > > <http://lists.fd.io> <sudhir=rtbrick....@lists.fd.io > > <mailto:rtbrick....@lists.fd.io> > wrote: > > > > > > Hi Ben, > > Thanks for the update. > > > > Regards, > > Sudhir > > > > On Thu, Feb 24, 2022 at 6:37 PM Benoit Ganne (bganne) > > <bga...@cisco.com <mailto:bga...@cisco.com> > wrote: > > > > > > Hi Sudhir, > > > > I am working on a few bugfixes for ASan. > > Right now, I'd recommend to use ASan in debug builds only, > > there are several issues that need to be fixed in release mode. > > > > Best > > ben > > > > > -----Original Message----- > > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of Sudhir > CR > > via > > > lists.fd.io <http://lists.fd.io> > > > Sent: jeudi 24 février 2022 14:03 > > > To: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > > Subject: [vpp-dev] VPP 2110 with AddressSanitizer enabled > > > > > > Hi Team, > > > > > > > > > We compiled vpp code with AddressSanitizer enabled. We > used > > the below > > > command to compile code. > > > > > > > > > sudo make rebuild-release VPP_EXTRA_CMAKE_ARGS=- > > > DVPP_ENABLE_SANITIZE_ADDR=ON > > > > > > > > > > > > we are starting vpp with below command > > > > > > > > > sudo ./vpp -c /etc/vpp/startup.conf > > > > > > > > > > > > But vpp startup is failed with below AddressSanitizer > error > > > > > > > > > AddressSanitizer:DEADLYSIGNAL > > > > > ================================================================= > > > ==1442028==ERROR: AddressSanitizer: SEGV on unknown > address > > 0x0200255e3b3e > > > (pc 0x7fe59ae338f7 bp 0x7ffc128b1c40 sp 0x7ffc128b1480 > T0) > > > ==1442028==The signal is caused by a READ memory access. > > > #0 0x7fe59ae338f6 in sanitizer_unpoison_push__ > > > > > /home/supervisor/development/libvpp/src/vppinfra/sanitizer.h:54:17 > > > #1 0x7fe59ae338f6 in hash_memory64 > > > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:157:15 > > > #2 0x7fe59ae338f6 in hash_memory > > > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:280:10 > > > #3 0x7fe59ae34b4e in key_sum > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c > > > #4 0x7fe59ae34b4e in lookup.llvm.8926505759877686271 > > > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:557:7 > > > #5 0x7fe59aeb203f in _hash_set3 > > > > > /home/supervisor/development/libvpp/src/vppinfra/hash.c:848:10 > > > #6 0x7fe59c4edc8e in config_one_plugin > > > > > /home/supervisor/development/libvpp/src/vlib/unix/plugin.c:710:3 > > > #7 0x7fe59c4edc8e in vlib_plugin_config > > > > > /home/supervisor/development/libvpp/src/vlib/unix/plugin.c:775:12 > > > #8 0x7fe59c49c3f6 in vlib_unix_main > > > > > /home/supervisor/development/libvpp/src/vlib/unix/main.c:764:12 > > > #9 0x55a040ce2e40 in main > > > > > /home/supervisor/development/libvpp/src/vpp/vnet/main.c:344:14 > > > #10 0x7fe59a29ebf6 in __libc_start_main /build/glibc- > > S9d2JN/glibc- > > > 2.27/csu/../csu/libc-start.c:310 > > > #11 0x55a040c41479 in _start > > > > (/home/supervisor/development/libvpp/build-root/install-vpp- > > > native/vpp/bin/vpp+0x41479) > > > > > > AddressSanitizer can not provide additional info. > > > SUMMARY: AddressSanitizer: SEGV > > > > > /home/supervisor/development/libvpp/src/vppinfra/sanitizer.h:54:17 in > > > sanitizer_unpoison_push__ > > > ==1442028==ABORTING > > > Aborted > > > > > > > > > > > > contents of startup.conf looks like below > > > > > > > > > > > > unix { > > > nodaemon > > > log /var/log/vpp/vpp.log > > > full-coredump > > > cli-listen localhost:5003 > > > runtime-dir /shm/run/vpp/ > > > } > > > > > > api-trace { > > > on > > > } > > > > > > cpu { > > > main-core 2 > > > corelist-workers 3,4 > > > } > > > > > > heapsize 600M > > > > > > statseg { > > > size 150M > > > } > > > plugin_path /usr/local/lib/vpp_plugins/ > > > plugins { > > > plugin rtbrick_bcm_plugin.so { disable } > > > plugin dpdk_plugin.so { disable } > > > } > > > > > > > > > > > > Please let me know any suggestions on how to resolve this > > Error. > > > > > > > > > Thanks in Advance, > > > > > > Sudhir > > > > > > > > > NOTICE TO RECIPIENT This e-mail message and any > attachments > > are > > > confidential and may be privileged. If you received this > e- > > mail in error, > > > any review, use, dissemination, distribution, or copying > of > > this e-mail is > > > strictly prohibited. Please notify us immediately of the > > error by return > > > e-mail and please delete this message from your system. > For > > more > > > information about Rtbrick, please visit us at > > www.rtbrick.com <http://www.rtbrick.com> > > > <http://www.rtbrick.com> > > > > > > > > NOTICE TO RECIPIENT This e-mail message and any attachments are > > confidential and may be privileged. If you received this e-mail in error, > > any review, use, dissemination, distribution, or copying of this e-mail > is > > strictly prohibited. Please notify us immediately of the error by return > > e-mail and please delete this message from your system. For more > > information about Rtbrick, please visit us at www.rtbrick.com > > <http://www.rtbrick.com> > > > > > > > > > > > > NOTICE TO RECIPIENT This e-mail message and any attachments are > > confidential and may be privileged. If you received this e-mail in error, > > any review, use, dissemination, distribution, or copying of this e-mail > is > > strictly prohibited. Please notify us immediately of the error by return > > e-mail and please delete this message from your system. For more > > information about Rtbrick, please visit us at www.rtbrick.com > > <http://www.rtbrick.com> > -- NOTICE TO RECIPIENT This e-mail message and any attachments are confidential and may be privileged. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of this e-mail is strictly prohibited. Please notify us immediately of the error by return e-mail and please delete this message from your system. For more information about Rtbrick, please visit us at www.rtbrick.com <http://www.rtbrick.com>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21031): https://lists.fd.io/g/vpp-dev/message/21031 Mute This Topic: https://lists.fd.io/mt/89364107/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
