Hi Rob! Thanks for the suggestion of double gdb.
With the native gdb, this gets me to: # gdb --args gdb Guitar GNU gdb (GDB) 15.1 Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64--netbsd". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from gdb... Reading symbols from /usr/libdata/debug//usr/bin/gdb.debug... (gdb) r Starting program: /usr/bin/gdb Guitar [New process 28508] [New LWP 27971 of process 28508] [New LWP 3691 of process 28508] [New LWP 25882 of process 28508] [New LWP 10964 of process 28508] [New LWP 2855 of process 28508] [New LWP 16996 of process 28508] [New LWP 23625 of process 28508] GNU gdb (GDB) 15.1 Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64--netbsd". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from Guitar... (gdb) r Starting program: /usr/pkg/bin/Guitar [New process 28508] ctype(3) isupper: invalid input: -61 Thread 10 "gdb worker" received signal SIGABRT, Aborted. [Switching to LWP 16996 of process 28508] 0x000075433d78b4da in _lwp_kill () from /usr/lib/libc.so.12 (gdb) bt #0 0x000075433d78b4da in _lwp_kill () from /usr/lib/libc.so.12 #1 0x000075433d796914 in abort () at /usr/src/lib/libc/stdlib/abort.c:74 #2 0x000075433d786c13 in ctype_nasaldemon (func=func@entry=0x75433d7b4098 <__func__.7> "isupper", c=<optimized out>) at /usr/src/lib/libc/gen/isctype.c:65 #3 0x000075433d787035 in ctype_check (c=<optimized out>, func=0x75433d7b4098 <__func__.7> "isupper") at /usr/src/lib/libc/gen/isctype.c:73 #4 isupper (c=<optimized out>) at /usr/src/lib/libc/gen/isctype.c:102 #5 0x000000000088bd82 in ada_decode[abi:cxx11](char const*, bool, bool, bool) () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/ada-lang.c:1563 #6 0x0000000000bdb235 in ada_language::sniff_from_mangled_name () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/ada-lang.c:13531 #7 0x00000000006d15f9 in symbol_find_demangled_name () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/symtab.c:961 #8 0x000000000072a0f0 in operator() () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/minsyms.c:1494 #9 0x0000000000ba76b1 in std::function<void()>::operator() () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/std_function.h:591 #10 std::__invoke_impl<void, std::function<void()>&> () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:61 #11 std::__invoke_r<void, std::function<void()>&> () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:111 #12 std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}::operator()() const () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1489 #13 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>::operator()() const () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1430 #14 std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&>(std::__invoke_other, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&) () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:61 #15 std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&>(std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&) () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:116 #16 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void> >::_M_invoke(std::_Any_data const&) () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/std_function.h:291 #17 0x0000000000ba762f in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>::operator() () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/std_function.h:591 #18 std::__future_base::_State_baseV2::_M_do_set () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:587 #19 0x000075433dcae711 in pthread_once (once_control=0x7542fffd0388, routine=0x75433db44bca <std::__once_proxy()>) at /usr/src/lib/libpthread/pthread_once.c:66 #20 0x0000000000ba7b1f in __gthread_once () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/gthr-default.h:703 #21 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*> () at /usr/obj/amd64.gcc.20250624/usr/include/g++/mutex:891 #22 0x0000000000a8bd99 in std::__future_base::_State_baseV2::_M_set_result () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:426 #23 std::__future_base::_Task_state<std::function<void()>, std::allocator<int>, void()>::_M_run () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1492 #24 std::packaged_task<void()>::operator() () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1626 #25 gdb::thread_pool::thread_function () at /usr/src/external/gpl3/gdb/lib/libgdbsupport/../../dist/gdbsupport/thread-pool.cc:245 #26 0x000075433db43d3b in std::execute_native_thread_routine (__p=0x75433d3c6d80) at /usr/src/external/gpl3/gcc/dist/libstdc++-v3/src/c++11/thread.cc:82 #27 0x000075433dcb32e1 in pthread__create_tramp (cookie=0x75433d3a9800) at /usr/src/lib/libpthread/pthread.c:605 #28 0x000075433d6706e0 in ?? () from /usr/lib/libc.so.12 #29 0x0000000000200000 in ?? () #30 0x0000000000000000 in ?? () (gdb) but I'm not sure how to interpret this - this looks like the backtrace of the inner gdb, not of Guitar. I'm also confused that it tries to decode ada. Guitar is a c/c++ applcation. Is there a way to get more information from the Guitar process here? Thread 9's backtrace looks more sane: Thread 9 (LWP 23625 of process 28508 "gdb worker"): #0 0x0000000000ada164 in d_print_comp () #1 0x0000000000adae55 in d_print_mod_list () #2 0x0000000000adabb5 in d_print_function_type () #3 0x0000000000ada169 in d_print_comp () #4 0x0000000000ad770e in d_print_comp_inner () --Type <RET> for more, q to quit, c to continue without paging-- #5 0x0000000000ada169 in d_print_comp () #6 0x0000000000adf248 in cplus_demangle_print_callback () #7 0x0000000000adf5f0 in d_demangle_callback () #8 0x0000000000adf750 in cplus_demangle_v3 () #9 0x0000000000ac772f in cplus_demangle () #10 0x0000000000a0a292 in bfd_demangle () #11 0x0000000000836b99 in gdb_demangle () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/cp-support.c:1640 #12 0x0000000000bce00a in cplus_language::sniff_from_mangled_name () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/c-lang.c:972 #13 0x00000000006d15f9 in symbol_find_demangled_name () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/symtab.c:961 #14 0x000000000072a0f0 in operator() () at /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/minsyms.c:1494 #15 0x0000000000ba76b1 in std::function<void()>::operator() () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/std_function.h:591 #16 std::__invoke_impl<void, std::function<void()>&> () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:61 #17 std::__invoke_r<void, std::function<void()>&> () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:111 #18 std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}::operator()() const () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1489 #19 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>::operator()() const () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1430 #20 std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&>(std::__invoke_other, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&) () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:61 #21 std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&>(std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>&) () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/invoke.h:116 #22 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void> >::_M_invoke(std::_Any_data const&) () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/std_function.h:291 #23 0x0000000000ba762f in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>::operator() () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/std_function.h:591 #24 std::__future_base::_State_baseV2::_M_do_set () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:587 #25 0x000075433dcae711 in pthread_once (once_control=0x754304276288, routine=0x75433db44bca <std::__once_proxy()>) at /usr/src/lib/libpthread/pthread_once.c:66 #26 0x0000000000ba7b1f in __gthread_once () at /usr/obj/amd64.gcc.20250624/usr/include/g++/bits/gthr-default.h:703 #27 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*> () at /usr/obj/amd64.gcc.20250624/usr/include/g++/mutex:891 #28 0x0000000000a8bd99 in std::__future_base::_State_baseV2::_M_set_result () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:426 #29 std::__future_base::_Task_state<std::function<void()>, std::allocator<int>, void()>::_M_run () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1492 #30 std::packaged_task<void()>::operator() () at /usr/obj/amd64.gcc.20250624/usr/include/g++/future:1626 #31 gdb::thread_pool::thread_function () at /usr/src/external/gpl3/gdb/lib/libgdbsupport/../../dist/gdbsupport/thread-pool.cc:245 #32 0x000075433db43d3b in std::execute_native_thread_routine (__p=0x75433d3c6da0) at /usr/src/external/gpl3/gcc/dist/libstdc++-v3/src/c++11/thread.cc:82 #33 0x000075433dcb32e1 in pthread__create_tramp (cookie=0x75433d3ab400) at /usr/src/lib/libpthread/pthread.c:605 #34 0x000075433d6706e0 in ?? () from /usr/lib/libc.so.12 #35 0x0000000000000000 in ?? () Thread 8 and 1 are in demangling-ADA mode too, thread 6 and 4 in demangling-c++ mode. Too many threads :-) When I try to quit this session, it fails badly: (gdb) q A debugging session is active. Inferior 1 [process 28508] will be killed. Quit anyway? (y or n) y [ 2469556.8125233] sorry, pid 11315 was killed: orphaned traced process /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/thread.c:252: internal-error: set_thread_exited: Assertion `nr_deleted == 1' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. ----- Backtrace ----- 0xa16c5b _Z22gdb_internal_backtracev 0x8d9f87 _ZL17internal_vproblemP16internal_problemPKciS2_P13__va_list_tag 0x8da2b7 _Z15internal_verrorPKciS0_P13__va_list_tag 0xcf3ca7 _Z18internal_error_locPKciS0_z 0x90885f _Z17set_thread_exitedP11thread_infoSt8optionalImEb 0x9d60e4 _ZN8inferior17clear_thread_listEv 0x9d6c8f _Z13exit_inferiorP8inferior 0x924293 _Z22generic_mourn_inferiorv 0xb68bc4 _ZN16inf_child_target14mourn_inferiorEv 0xb685bd _ZN17inf_ptrace_target14mourn_inferiorEv 0xb67aa6 _ZN17inf_ptrace_target4killEv 0x9111b5 _Z11target_killv 0x90211f _Z10quit_forcePii 0xb1495e _Z12quit_commandPKci 0xb32555 _Z8cmd_funcP16cmd_list_elementPKci 0x901afa _Z15execute_commandPKci 0xa15138 _Z15command_handlerPKc 0xa16847 _Z20command_line_handlerOSt10unique_ptrIcN3gdb13xfree_deleterIcEEE 0xa15a2d _ZL23gdb_rl_callback_handlerPc 0xd64236 rl_callback_read_char 0xa15bb6 _ZL42gdb_rl_callback_read_char_wrapper_noexceptv 0xa15e0e _ZL33gdb_rl_callback_read_char_wrapperPv 0x8dce09 _ZL19stdin_event_handleriPv 0xceddc5 _ZL18gdb_wait_for_eventi.part.0 0xcee4ef _Z16gdb_do_one_eventi 0x9968dd _ZL21captured_command_loopv 0x999277 _Z8gdb_mainP18captured_main_args 0xdd32d7 main --------------------- /usr/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/thread.c:252: internal-error: set_thread_exited: Assertion `nr_deleted == 1' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Some readline problem? When I let it create a core dump, the backtrace is mostly useless: (gdb) bt #0 0x00007c9e47e924da in ?? () #1 0x00007c9e47e9d914 in ?? () #2 0xffffffffffffffdf in ?? () #3 0xffffffffffffffff in ?? () #4 0x0000000000000246 in ?? () #5 0x00007c9e3cf28000 in ?? () #6 0x00007f7fffc8b9a0 in ?? () #7 0x00000000008d811b in extension_language_python () #8 0x7fffffffffffffff in ?? () #9 0x7fffffffffffffff in ?? () #10 0x00007f7fffc8ba70 in ?? () #11 0x00000000008da0f5 in Suffix3DNow () #12 0x00000000011afa68 in ?? () #13 0x00007f7fffc8ba00 in ?? () #14 0x000000fcffc8b9e0 in ?? () #15 0x00007c9e3cf28000 in ?? () #16 0x00007c9e47f04d38 in ?? () #17 0x0000000100000002 in ?? () #18 0x00007c9e3cf28000 in ?? () #19 0x00000000000000fe in ?? () #20 0x00000000000000fe in ?? () #21 0x0000000000000000 in ?? () You also suggested using devel/gdb but that doesn't get me far at all: # /usr/pkg/bin/gdb Guitar GNU gdb (GDB) 10.1 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64--netbsd". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from Guitar... (gdb) r Starting program: /usr/pkg/bin/Guitar /scratch/devel/gdb/work/gdb-10.1/gdb/thread.c:1309: internal-error: void switch_to_thread(thread_info*): Assertion `thr != NULL' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n) A core dump of that has a backtrace that looks completely wrong, including ... #4 0x0000000001e82cb8 in sh_linux_sigreturn_tramp_frame () #5 0x000071965060fdc0 in ?? () #6 0x00007f7fff9d9fc0 in ?? () #7 0x00000000018aca4c in m32c_cgen_insn_opcode_table () #8 0x7fffffffffffffff in ?? () #9 0x7fffffffffffffff in ?? () #10 0x0000000001e8edaa in aarch64_opcode_table () #11 0x2dc6e5548f51cd07 in ?? () #12 0x00007f7fff9da090 in ?? () #13 0x00000000018b27ce in m32c_cgen_insn_opcode_table () ... > (gdb) bt > #0 0x000074897bcec4da in _lwp_kill () from /usr/lib/libc.so.12 > #1 0x000074897bcf7914 in abort () > at /usr/netbsd/current/src/lib/libc/stdlib/abort.c:74 > #2 0x000074897bce7c13 in ctype_nasaldemon ( > func=func@entry=0x74897bd13718 <__func__.7> "isupper", c=<optimized out>) > at /usr/netbsd/current/src/lib/libc/gen/isctype.c:65 > #3 0x000074897bce8035 in ctype_check (c=<optimized out>, > func=0x74897bd13718 <__func__.7> "isupper") > at /usr/netbsd/current/src/lib/libc/gen/isctype.c:73 > #4 isupper (c=<optimized out>) > at /usr/netbsd/current/src/lib/libc/gen/isctype.c:102 > #5 0x0000000000fbed82 in ada_decode[abi:cxx11](char const*, bool, bool, bool) > () > at > /usr/netbsd/current/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/ada-lang.c:1563 > #6 0x000000000130e235 in ada_language::sniff_from_mangled_name () > at > /usr/netbsd/current/src/external/gpl3/gdb/lib/libgdb/../../dist/gdb/ada-lang.c:13531 Ah you have the same ada demangler in the vlc backtrace! I still wonder why we get there at all but it looks like the ada demangler code has a ctype(3) issue too. Cheers, Thomas
