Hi All, Just suffered 3 bareos-dir crashes within a week.
$ bareos-dir --version 24.0.2~pre41.988426cda Backtrace is below. A null pointer dereference occurred inside glibc at pthread_kill.c:50 <https://github.com/bminor/glibc/blob/master/nptl/pthread_kill.c#L50>. Looks like bareos-dir passed an invalid thread_id to pthread_kill(). >From the backtrace it looks like it crashed when the job was cancelled. I have browsed Bareos Git commit log and can see the following very interesting commit message <https://github.com/bareos/bareos/commit/1d8529a342dbf3f3ae327f8b7bb3ded97bb2ef33>: "...if you try to cancel a job while that job is in the process of ending, then you risk an unfixable crash." But this is just my guess. Could you please have a look at the backtrace below and advise if that crash is indeed caused by this "unfixable" bug or is it a different issue that should be investigated? This commit mentions that these crashes are unlikely, but I have already seen three ones within a week. (gdb) bt #0 0x00007fd29888b9b3 in __pthread_kill_implementation (threadid=140541438182976, signo=12, no_tid=0) at pthread_kill.c:50 #1 0x00007fd298f540de in JobControlRecord::MyThreadSendSignal (this=this@entry=0x56047acc47f0, sig=sig@entry=12) at ../../../../../../../bareos_PR-2177/BUILD/bareos/cmake-build/core/src/lib/../../../../core/src/lib/jcr.cc:440 #2 0x00005604799b4d1e in directordaemon::CancelFileDaemonJob (jcr=0x56047acc47f0, ua=0x7fd2480160a0) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/fd_cmds.cc:988 #3 directordaemon::CancelJob(directordaemon::UaContext*, JobControlRecord*) [clone .isra.0] (ua=0x7fd2480160a0, jcr=0x56047acc47f0) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/job.cc:696 #4 0x0000560479944dbd in directordaemon::CancelJobs (ua=0x7fd2480160a0) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/ua_cmds.cc:791 #5 directordaemon::CancelCmd (ua=0x7fd2480160a0, cmd=<optimized out>) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/ua_cmds.cc:810 #6 0x000056047994bfaf in directordaemon::UaContext::execute (cmdstruct=<optimized out>, this=0x7fd2480160a0) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/ua_cmds.cc:534 #7 directordaemon::Do_a_command (ua=0x7fd2480160a0) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/ua_cmds.cc:585 #8 0x000056047993a242 in directordaemon::HandleUserAgentClientRequest (user_agent_socket=0x7fd290000b60) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/ua_server.cc:106 #9 directordaemon::HandleConnectionRequest (config=<optimized out>, arg=0x7fd290000b60) at ../../../../../../../bareos-testing_bareos-24/BUILD/bareos/cmake-build/core/src/dird/../../../../core/src/dird/socket_server.cc:116 #10 0x00007fd298f6a7d2 in std::function<void* (ConfigurationParser*, void*)>::operator()(ConfigurationParser*, void*) const (__args#1=0x7fd290000b60, __args#0=<optimized out>, this=0x7fd290001ff8) at /usr/include/c++/11/bits/std_function.h:590 #11 WorkerThread (l=..., ThreadInvokedHandler=..., config=0x56047ab96160, data=0x7fd290000b60, run_condition=...) at ../../../../../../../bareos_PR-2177/BUILD/bareos/cmake-build/core/src/lib/../../../../core/src/lib/thread_list.cc:201 #12 0x00007fd298f64375 in std::__invoke_impl<void, void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> >(std::__invoke_other, void (*&&)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>&&, std::function<void* (ConfigurationParser*, void*)>&&, ConfigurationParser*&&, void*&&, std::shared_ptr<IsRunningCondition>&&) ( __f=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:61 #13 std::__invoke<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> >(void (*&&)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>&&, std::function<void* (ConfigurationParser*--Type <RET> for more, q to quit, c to continue without paging--c , void*)>&&, ConfigurationParser*&&, void*&&, std::shared_ptr<IsRunningCondition>&&) (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96 #14 std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>) (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259 #15 std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > >::operator()() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266 #16 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > > >::_M_run() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211 #17 0x00007fd298cdbad4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #18 0x00007fd298889d22 in start_thread (arg=<optimized out>) at pthread_create.c:443 #19 0x00007fd29890ed40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Kind regard, Grigory Trenin -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/bareos-users/c78feb48-ca47-4d51-8722-1358b3e49ad1n%40googlegroups.com.
