Hi Stanilas, Yes there is a constant stream of fixes for AddressSanitizer as it is not in CI/CD yet. I'd suggest you try to reproduce your issue with VPP master instead, because the initial crash you reported might have been fixed in master. If it is fixed, you can then git bisect to find the relevant fix. If not, ASan should work on master.
Best ben > -----Original Message----- > From: Stanislav Zaikin <zsta...@gmail.com> > Sent: jeudi 21 octobre 2021 19:26 > To: Benoit Ganne (bganne) <bga...@cisco.com> > Cc: vpp-dev <vpp-dev@lists.fd.io> > Subject: Re: [vpp-dev] assert in pool_elt_at_index > > Hi Benoit, > I've tried but got this crash. Maybe some fixes were merged to handle this > issue? (I'm using 21.01 + additional patches from the master) > > > Program received signal SIGSEGV, Segmentation fault. > 0x000000000042c037 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, > unsigned long*, unsigned long*) () > (gdb) bt > #0 0x000000000042c037 in __asan::FakeStack::AddrIsInFakeStack(unsigned > long, unsigned long*, unsigned long*) () > #1 0x00000000004a31b4 in > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) > () > #2 0x00000000004b2f4a in > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool > (*)(__sanitizer::ThreadContextBase*, void*), void*) () > #3 0x00000000004a30eb in __asan::FindThreadByStackAddress(unsigned long) > () > #4 0x0000000000429265 in > __asan::AddressDescription::AddressDescription(unsigned long, unsigned > long, bool) () > #5 0x000000000042adb3 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, > unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned > long) () > #6 0x000000000049cd81 in __asan::ReportGenericError(unsigned long, > unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned > int, bool) () > #7 0x000000000049d5e8 in __asan_report_load4 () > #8 0x00007ffff4fc18e8 in clib_march_select_fn_ptr_by_name > (r=0x7ffff7858740 <vnet_interface_output_node_march_fn_registration>, > name=0x7ffff6fd5980 <.str> "hsw") at /home/vpp/vpp/src/vppinfra/cpu.h:94 > #9 0x00007ffff4fc17a1 in vnet_interface_output_node_get > (vm=0x7ffff3a2d340 <vlib_global_main>) at > /home/vpp/vpp/src/vnet/interface_output.c:538 > #10 0x00007ffff4f152cd in vnet_register_interface (vnm=0x7ffff7a6b320 > <vnet_main>, dev_class_index=34, dev_instance=0, hw_class_index=31, > hw_instance=0) at /home/vpp/vpp/src/vnet/interface.c:810 > #11 0x00007ffff4ff34ca in vnet_main_init (vm=0x7ffff3a2d340 > <vlib_global_main>) at /home/vpp/vpp/src/vnet/misc.c:81 > #12 0x00007ffff353d476 in call_init_exit_functions_internal > (vm=0x7ffff3a2d340 <vlib_global_main>, headp=0x7ffff3a2d998 > <vlib_global_main+1624>, call_once=1, do_sort=1) at > /home/vpp/vpp/src/vlib/init.c:350 > #13 0x00007ffff353d258 in vlib_call_init_exit_functions (vm=0x7ffff3a2d340 > <vlib_global_main>, headp=0x7ffff3a2d998 <vlib_global_main+1624>, > call_once=1) at /home/vpp/vpp/src/vlib/init.c:364 > #14 0x00007ffff353d561 in vlib_call_all_init_functions (vm=0x7ffff3a2d340 > <vlib_global_main>) at /home/vpp/vpp/src/vlib/init.c:386 > #15 0x00007ffff35c8711 in vlib_main (vm=0x7ffff3a2d340 <vlib_global_main>, > input=0x7fff9cb4bec0) at /home/vpp/vpp/src/vlib/main.c:2213 > #16 0x00007ffff376b808 in thread0 (arg=140737280922432) at > /home/vpp/vpp/src/vlib/unix/main.c:670 > #17 0x00007ffff269204c in clib_calljmp () at > /home/vpp/vpp/src/vppinfra/longjmp.S:123 > #18 0x00007fffffffc980 in ?? () > #19 0x00007ffff376ad55 in vlib_unix_main (argc=2, argv=0x7fffffffe498) at > /home/vpp/vpp/src/vlib/unix/main.c:747 > #20 0x00000000004c8fa8 in main (argc=2, argv=0x7fffffffe498) at > /home/vpp/vpp/src/vpp/vnet/main.c:338 > (gdb) > > > On Mon, 18 Oct 2021 at 14:29, Benoit Ganne (bganne) <bga...@cisco.com > <mailto:bga...@cisco.com> > wrote: > > > You can try running with AddressSanitizer: > https://fd.io/docs/vpp/master/troubleshooting/sanitizer.html#id2 > That could catch the corruption earlier with more clues. > > Best > ben > > > -----Original Message----- > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> <vpp- > d...@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of Stanislav > > Zaikin > > Sent: vendredi 15 octobre 2021 14:54 > > To: vpp-dev <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > > Subject: Re: [vpp-dev] assert in pool_elt_at_index > > > > Any thoughts will be much appreciated. This bug is quite > reproducible in > > my environment, but I don't know what else to check. > > > > > > On Thu, 14 Oct 2021 at 08:51, Stanislav Zaikin via lists.fd.io > <http://lists.fd.io> > > <http://lists.fd.io> <zstaseg=gmail....@lists.fd.io > <mailto:gmail....@lists.fd.io> > > <mailto:gmail....@lists.fd.io <mailto:gmail....@lists.fd.io> > > > wrote: > > > > > > Hi Florin, > > Hi Rajith, > > > > It shouldn't be the pool expansion case, I have > > 8341f76fd1cd4351961cd8161cfed2814fc55103. > > Moreover, in this case _e would be different from > > &load_balance_pool[3604]. I've found some of those expansions (in > other > > places), in those cases a pointer to the element has a different > address. > > > > > > > > > > On Thu, 14 Oct 2021 at 06:45, Rajith PR <raj...@rtbrick.com > <mailto:raj...@rtbrick.com> > > <mailto:raj...@rtbrick.com <mailto:raj...@rtbrick.com> > > wrote: > > > > > > HI Stanislav, > > > > My guess is you don't have the commit below. > > > > commit 8341f76fd1cd4351961cd8161cfed2814fc55103 > > Author: Dave Barach <d...@barachs.net > <mailto:d...@barachs.net> > > <mailto:d...@barachs.net <mailto:d...@barachs.net> > > > > Date: Wed Jun 3 08:05:15 2020 -0400 > > > > fib: add barrier sync, pool/vector expand cases > > > > load_balance_alloc_i(...) is not thread safe > when the > > load_balance_pool or combined counter vectors > expand. > > > > Type: fix > > > > Signed-off-by: Dave Barach <d...@barachs.net > <mailto:d...@barachs.net> > > <mailto:d...@barachs.net <mailto:d...@barachs.net> > > > > Change-Id: > I7f295ed77350d1df0434d5ff461eedafe79131de > > > > > > Thanks, > > Rajith > > > > On Thu, Oct 14, 2021 at 3:57 AM Florin Coras > > <fcoras.li...@gmail.com <mailto:fcoras.li...@gmail.com> > <mailto:fcoras.li...@gmail.com <mailto:fcoras.li...@gmail.com> > > wrote: > > > > > > Hi Stanislav, > > > > The only thing I can think of is that main > thread grows > > the pool, or the pool’s bitmap, without a worker barrier while the > worker > > that asserts is trying to access it. Is main thread busy doing > something > > (e.g., adding routes/interfaces) when the assert happens? > > > > Regards, > > Florin > > > > > > > > On Oct 13, 2021, at 2:52 PM, > Stanislav Zaikin > > <zsta...@gmail.com <mailto:zsta...@gmail.com> > <mailto:zsta...@gmail.com <mailto:zsta...@gmail.com> > > wrote: > > > > Hi Florin, > > > > I wasn't aware of those helper > functions, thanks! > > But yeah, it also returns 0 (sorry, but there's the trace of > another > > crash) > > > > Thread 3 "vpp_wk_0" received signal > SIGABRT, > > Aborted. > > [Switching to Thread 0x7f9cc0f6a700 > (LWP 3546)] > > __GI_raise (sig=sig@entry=6) at > > ../sysdeps/unix/sysv/linux/raise.c:51 > > 51 > ../sysdeps/unix/sysv/linux/raise.c: No such > > file or directory. > > (gdb) bt > > #0 __GI_raise (sig=sig@entry=6) at > > ../sysdeps/unix/sysv/linux/raise.c:51 > > #1 0x00007f9d61542921 in __GI_abort > () at > > abort.c:79 > > #2 0x00007f9d624da799 in os_panic > () at > > /home/vpp/vpp/src/vppinfra/unix-misc.c:177 > > #3 0x00007f9d62420f49 in debugger > () at > > /home/vpp/vpp/src/vppinfra/error.c:84 > > #4 0x00007f9d62420cc7 in > _clib_error > > (how_to_die=2, function_name=0x0, line_number=0, > fmt=0x7f9d644348d0 "%s:%d > > (%s) assertion `%s' fails") at > /home/vpp/vpp/src/vppinfra/error.c:143 > > #5 0x00007f9d636695b4 in > load_balance_get > > (lbi=4569) at /home/vpp/vpp/src/vnet/dpo/load_balance.h:222 > > #6 0x00007f9d63668247 in > mpls_lookup_node_fn_hsw > > (vm=0x7f9ceb0138c0, node=0x7f9ceee6f700, > from_frame=0x7f9cef9c9240) at > > /home/vpp/vpp/src/vnet/mpls/mpls_lookup.c:229 > > #7 0x00007f9d63008076 in > dispatch_node > > (vm=0x7f9ceb0138c0, node=0x7f9ceee6f700, > type=VLIB_NODE_TYPE_INTERNAL, > > dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x7f9cef9c9240, > > last_time_stamp=1837178878370487) at > /home/vpp/vpp/src/vlib/main.c:1217 > > #8 0x00007f9d630089e7 in > dispatch_pending_node > > (vm=0x7f9ceb0138c0, pending_frame_index=2, > > last_time_stamp=1837178878370487) at > /home/vpp/vpp/src/vlib/main.c:1376 > > #9 0x00007f9d63002441 in > vlib_main_or_worker_loop > > (vm=0x7f9ceb0138c0, is_main=0) at > /home/vpp/vpp/src/vlib/main.c:1904 > > #10 0x00007f9d630012e7 in > vlib_worker_loop > > (vm=0x7f9ceb0138c0) at /home/vpp/vpp/src/vlib/main.c:2038 > > #11 0x00007f9d6305995d in > vlib_worker_thread_fn > > (arg=0x7f9ce1b88540) at /home/vpp/vpp/src/vlib/threads.c:1868 > > #12 0x00007f9d62445214 in > clib_calljmp () at > > /home/vpp/vpp/src/vppinfra/longjmp.S:123 > > #13 0x00007f9cc0f69c90 in ?? () > > #14 0x00007f9d63051b83 in > > vlib_worker_thread_bootstrap_fn (arg=0x7f9ce1b88540) at > > /home/vpp/vpp/src/vlib/threads.c:585 > > #15 0x00007f9cda360355 in > eal_thread_loop > > (arg=0x0) at ../src-dpdk/lib/librte_eal/linux/eal_thread.c:127 > > #16 0x00007f9d629246db in > start_thread > > (arg=0x7f9cc0f6a700) at pthread_create.c:463 > > #17 0x00007f9d6162371f in clone () > at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > (gdb) select 5 > > (gdb) print pifi( load_balance_pool, > 4569 ) > > $1 = 0 > > (gdb) source > ~/vpp/extras/gdb/gdbinit > > Loading vpp functions... > > Load vlLoad pe > > Load pifi > > Load node_name_from_index > > Load vnet_buffer_opaque > > Load vnet_buffer_opaque2 > > Load bitmap_get > > Done loading vpp functions... > > (gdb) pifi load_balance_pool 4569 > > pool_is_free_index > (load_balance_pool, 4569)$2 = 0 > > > > > > On Wed, 13 Oct 2021 at 21:55, Florin > Coras > > <fcoras.li...@gmail.com <mailto:fcoras.li...@gmail.com> > <mailto:fcoras.li...@gmail.com <mailto:fcoras.li...@gmail.com> > > wrote: > > > > > > Hi Stanislav, > > > > Just to make sure the gdb > macro is okay, > > could you run from gdb: pifi(pool, index)? The function is defined > in > > gdb_funcs.c. > > > > Regards, > > Florin > > > > > > On Oct 13, 2021, at 11:30 > AM, Stanislav > > Zaikin <zsta...@gmail.com <mailto:zsta...@gmail.com> > <mailto:zsta...@gmail.com <mailto:zsta...@gmail.com> > > wrote: > > > > Hello folks, > > > > I'm facing a strange issue > with 2 worker > > threads. Sometimes I get a crash either in "ip6-lookup" or "mpls- > lookup" > > nodes. They happen with assert in the pool_elt_at_index macro and > always > > inside the "load_balance_get" function. But the load_balance dpo > looks > > perfectly good, I mean it still has a lock and on regular deletion > (in the > > case when the load_balance dpo is deleted) it should be erased > properly > > (with dpo_reset). It happens usually when the main core is > executing > > vlib_worker_thread_barrier_sync_int(), and the other worker is > executing > > vlib_worker_thread_barrier_check(). > > And the strangest thing is, > when I run the > > vpp's gdb helper for checking "pool_index_is_free" or pifi, it > shows me > > that the index isn't free (and the macro in that case shouldn't > fire). > > > > > > Any thoughts and inputs are > appreciated. > > > > > > Thread 3 "vpp_wk_0" received > signal SIGABRT, > > Aborted. > > [Switching to Thread > 0x7fb4f2e22700 (LWP > > 3244)] > > __GI_raise (sig=sig@entry=6) > at > > ../sysdeps/unix/sysv/linux/raise.c:51 > > 51 > ../sysdeps/unix/sysv/linux/raise.c: No > > such file or directory. > > (gdb) bt > > #0 __GI_raise > (sig=sig@entry=6) at > > ../sysdeps/unix/sysv/linux/raise.c:51 > > #1 0x00007fb5933fa921 in > __GI_abort () at > > abort.c:79 > > #2 0x00007fb594392799 in > os_panic () at > > /home/vpp/vpp/src/vppinfra/unix-misc.c:177 > > #3 0x00007fb5942d8f49 in > debugger () at > > /home/vpp/vpp/src/vppinfra/error.c:84 > > #4 0x00007fb5942d8cc7 in > _clib_error > > (how_to_die=2, function_name=0x0, line_number=0, > fmt=0x7fb5962ec8d0 "%s:%d > > (%s) assertion `%s' fails") at > /home/vpp/vpp/src/vppinfra/error.c:143 > > #5 0x00007fb5954bd694 in > load_balance_get > > (lbi=3604) at /home/vpp/vpp/src/vnet/dpo/load_balance.h:222 > > #6 0x00007fb5954bc070 in > ip6_lookup_inline > > (vm=0x7fb51ceccd00, node=0x7fb520f6b700, frame=0x7fb52128e4c0) at > > /home/vpp/vpp/src/vnet/ip/ip6_forward.h:117 > > #7 0x00007fb5954bbdd5 in > > ip6_lookup_node_fn_hsw (vm=0x7fb51ceccd00, node=0x7fb520f6b700, > > frame=0x7fb52128e4c0) at > /home/vpp/vpp/src/vnet/ip/ip6_forward.c:736 > > #8 0x00007fb594ec0076 in > dispatch_node > > (vm=0x7fb51ceccd00, node=0x7fb520f6b700, > type=VLIB_NODE_TYPE_INTERNAL, > > dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x7fb52128e4c0, > > last_time_stamp=1808528151240447) at > /home/vpp/vpp/src/vlib/main.c:1217 > > #9 0x00007fb594ec09e7 in > > dispatch_pending_node (vm=0x7fb51ceccd00, pending_frame_index=5, > > last_time_stamp=1808528151240447) at > /home/vpp/vpp/src/vlib/main.c:1376 > > #10 0x00007fb594eba441 in > > vlib_main_or_worker_loop (vm=0x7fb51ceccd00, is_main=0) at > > /home/vpp/vpp/src/vlib/main.c:1904 > > #11 0x00007fb594eb92e7 in > vlib_worker_loop > > (vm=0x7fb51ceccd00) at /home/vpp/vpp/src/vlib/main.c:2038 > > #12 0x00007fb594f1195d in > > vlib_worker_thread_fn (arg=0x7fb513a48100) at > > /home/vpp/vpp/src/vlib/threads.c:1868 > > #13 0x00007fb5942fd214 in > clib_calljmp () at > > /home/vpp/vpp/src/vppinfra/longjmp.S:123 > > #14 0x00007fb4f2e21c90 in ?? > () > > #15 0x00007fb594f09b83 in > > vlib_worker_thread_bootstrap_fn (arg=0x7fb513a48100) at > > /home/vpp/vpp/src/vlib/threads.c:585 > > #16 0x00007fb50c218355 in > eal_thread_loop > > (arg=0x0) at ../src-dpdk/lib/librte_eal/linux/eal_thread.c:127 > > #17 0x00007fb5947dc6db in > start_thread > > (arg=0x7fb4f2e22700) at pthread_create.c:463 > > #18 0x00007fb5934db71f in > clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > (gdb) select 5 > > (gdb) print _e > > $1 = (load_balance_t *) > 0x7fb52651e580 > > (gdb) print > load_balance_pool[3604] > > $2 = {cacheline0 = > 0x7fb52651e580 "\001", > > lb_n_buckets = 1, lb_n_buckets_minus_1 = 0, lb_proto = > DPO_PROTO_IP6, > > lb_flags = LOAD_BALANCE_FLAG_NONE, lb_fib_entry_flags = > > (FIB_ENTRY_FLAG_CONNECTED | FIB_ENTRY_FLAG_LOCAL), lb_locks = 1, > lb_map = > > 4294967295, lb_urpf = 4094, lb_hash_config = 31, lb_buckets = 0x0, > > lb_buckets_inline = > {{{{dpoi_type = > > DPO_RECEIVE, dpoi_proto = DPO_PROTO_IP6, dpoi_next_node = 2, > dpoi_index = > > 2094}, as_u64 = 8993661649164}}, {{{dpoi_type = DPO_FIRST, > dpoi_proto = > > DPO_PROTO_IP4, dpoi_next_node = 0, dpoi_index = 0}, as_u64 = 0}}, > > {{{dpoi_type = DPO_FIRST, dpoi_proto = DPO_PROTO_IP4, > > dpoi_next_node = > 0, dpoi_index = > > 0}, as_u64 = 0}}, {{{dpoi_type = DPO_FIRST, dpoi_proto = > DPO_PROTO_IP4, > > dpoi_next_node = 0, dpoi_index = 0}, as_u64 = 0}}}} > > (gdb) print > &load_balance_pool[3604] > > $3 = (load_balance_t *) > 0x7fb52651e580 > > (gdb) source > ~/vpp/extras/gdb/gdbinit > > Loading vpp functions... > > Load vlLoad pe > > Load pifi > > Load node_name_from_index > > Load vnet_buffer_opaque > > Load vnet_buffer_opaque2 > > Load bitmap_get > > Done loading vpp > functions... > > (gdb) pifi load_balance_pool > 3604 > > pool_is_free_index > (load_balance_pool, > > 3604)$4 = 0 > > (gdb) info threads > > Id Target Id > Frame > > 1 Thread 0x7fb596bd2c40 > (LWP 727) > > "vpp_main" 0x00007fb594f1439b in clib_time_now_internal > (c=0x7fb59517ccc0 > > <vlib_global_main>, n=1808528155236639) at > > /home/vpp/vpp/src/vppinfra/time.h:215 > > 2 Thread 0x7fb4f3623700 > (LWP 2976) > > "eal-intr-thread" 0x00007fb5934dba47 in epoll_wait (epfd=17, > > events=0x7fb4f3622d80, maxevents=1, timeout=-1) at > > ../sysdeps/unix/sysv/linux/epoll_wait.c:30 > > * 3 Thread 0x7fb4f2e22700 > (LWP 3244) > > "vpp_wk_0" __GI_raise (sig=sig@entry=6) at > > ../sysdeps/unix/sysv/linux/raise.c:51 > > 4 Thread 0x7fb4f2621700 > (LWP 3246) > > "vpp_wk_1" 0x00007fb594ebf897 in vlib_worker_thread_barrier_check > () at > > /home/vpp/vpp/src/vlib/threads.h:439 > > > > > > -- > > > > Best regards > > Stanislav Zaikin > > > > > > > > > > > > > > > > -- > > > > Best regards > > Stanislav Zaikin > > > > > > > > > > > > > > NOTICE TO RECIPIENT This e-mail message and any > attachments > > are confidential and may be privileged. If you received this e- > mail in > > error, any review, use, dissemination, distribution, or copying of > this e- > > mail is strictly prohibited. Please notify us immediately of the > error by > > return e-mail and please delete this message from your system. For > more > > information about Rtbrick, please visit us at www.rtbrick.com > <http://www.rtbrick.com> > > <http://www.rtbrick.com> > > > > > > > > -- > > > > Best regards > > Stanislav Zaikin > > > > > > > > > > > > > > -- > > > > Best regards > > Stanislav Zaikin > > > > > -- > > Best regards > Stanislav Zaikin
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20355): https://lists.fd.io/g/vpp-dev/message/20355 Mute This Topic: https://lists.fd.io/mt/86295132/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
