[
https://issues.apache.org/jira/browse/KUDU-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795306#comment-17795306
]
Xixu Wang commented on KUDU-3524:
---------------------------------
This case only happens in macOS, see
src/kudu/util/notification.h#WaitForNotification()
> The TestScannerKeepAlivePeriodicallyCrossServers scenario fails with SIGABRT
> ----------------------------------------------------------------------------
>
> Key: KUDU-3524
> URL: https://issues.apache.org/jira/browse/KUDU-3524
> Project: Kudu
> Issue Type: Bug
> Reporter: Alexey Serbin
> Priority: Major
>
> Running the newly added tests scenario
> {{TestScannerKeepAlivePeriodicallyCrossServers}} fails with SIGABRT when run
> as the following on macOS (but I guess it's not macOS-specific) in DEBUG
> build:
> {noformat}
> ./bin/client-test --stress_cpu_threads=32
> --gtest_filter='*TestScannerKeepAlivePeriodicallyCrossServers*'
> {noformat}
> The error message and the stacktrace is below:
> {noformat}
> F20231113 12:21:13.431455 41195482 thread_restrictions.cc:79] Check failed:
> LoadTLS()->wait_allowed Waiting is not allowed to be used on this thread to
> prevent server-wide latency aberrations and deadlocks. Thread 41195482 (name:
> "rpc reactor", category: "reactor")
> *** Check failure stack trace: ***
> Process 77090 stopped
> * thread #335, name = 'rpc reactor-41195482', stop reason = signal SIGABRT
> frame #0: 0x00007fff205b890e libsystem_kernel.dylib`__pthread_kill + 10
> libsystem_kernel.dylib`__pthread_kill:
> -> 0x7fff205b890e <+10>: jae 0x7fff205b8918 ; <+20>
> 0x7fff205b8910 <+12>: movq %rax, %rdi
> 0x7fff205b8913 <+15>: jmp 0x7fff205b2ab9 ; cerror_nocancel
> 0x7fff205b8918 <+20>: retq
> Target 0: (client-test) stopped.
> (lldb) bt
> * thread #335, name = 'rpc reactor-41195482', stop reason = signal SIGABRT
> * frame #0: 0x00007fff205b890e libsystem_kernel.dylib`__pthread_kill + 10
> frame #1: 0x00007fff205e75bd libsystem_pthread.dylib`pthread_kill + 263
> frame #2: 0x00007fff2053c406 libsystem_c.dylib`abort + 125
> frame #3: 0x000000010f64ebd8
> libglog.1.dylib`google::LogMessage::SendToLog() [inlined]
> google::LogMessage::Fail() at logging.cc:1946:3 [opt]
> frame #4: 0x000000010f64ebd2
> libglog.1.dylib`google::LogMessage::SendToLog(this=0x000070001a95e108) at
> logging.cc:1920:5 [opt]
> frame #5: 0x000000010f64f47a
> libglog.1.dylib`google::LogMessage::Flush(this=0x000070001a95e108) at
> logging.cc:1777:5 [opt]
> frame #6: 0x000000010f65428f
> libglog.1.dylib`google::LogMessageFatal::~LogMessageFatal(this=0x000070001a95e108)
> at logging.cc:2557:5 [opt]
> frame #7: 0x000000010f650349
> libglog.1.dylib`google::LogMessageFatal::~LogMessageFatal(this=<unavailable>)
> at logging.cc:2556:37 [opt]
> frame #8: 0x000000010e545473
> libkudu_util.dylib`kudu::ThreadRestrictions::AssertWaitAllowed() at
> thread_restrictions.cc:79:3
> frame #9: 0x000000010013ebb9
> client-test`kudu::CountDownLatch::Wait(this=0x000070001a95e2a0) const at
> countdown_latch.h:74:5
> frame #10: 0x000000010a1749f5
> libkrpc.dylib`kudu::Notification::WaitForNotification(this=0x000070001a95e2a0)
> const at notification.h:127:12
> frame #11: 0x000000010a1748e9
> libkrpc.dylib`kudu::rpc::Proxy::SyncRequest(this=0x000000011317e9b8,
> method="ScannerKeepAlive", req=0x000070001a95e428, resp=0x000070001a95e408,
> controller=0x000070001a95e458) at proxy.cc:259:8
> frame #12: 0x000000010697220f
> libtserver_service_proto.dylib`kudu::tserver::TabletServerServiceProxy::ScannerKeepAlive(this=0x000000011317e9b8,
> req=0x000070001a95e428, resp=0x000070001a95e408,
> controller=0x000070001a95e458) at tserver_service.proxy.cc:98:10
> frame #13: 0x000000010525c5b6
> libkudu_client.dylib`kudu::client::KuduScanner::Data::KeepAlive(this=0x000000011290c700)
> at scanner-internal.cc:664:3
> frame #14: 0x0000000105269e76
> libkudu_client.dylib`kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(this=0x0000000112899858)::$_0::operator()()
> const at scanner-internal.cc:112:16
> frame #15: 0x0000000105269e30
> libkudu_client.dylib`decltype(__f=0x0000000112899858)::$_0&>(fp)())
> std::__1::__invoke<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
> long long,
> std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&>(kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
> long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&) at
> type_traits:3694:1
> frame #16: 0x0000000105269dd1 libkudu_client.dylib`void
> std::__1::__invoke_void_return_wrapper<void,
> true>::__call<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(__args=0x0000000112899858)::$_0&>(kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
> long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&) at
> __functional_base:348:9
> frame #17: 0x0000000105269d9d
> libkudu_client.dylib`std::__1::__function::__alloc_func<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
> long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0,
> std::__1::allocator<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
> long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0>, void
> ()>::operator(this=0x0000000112899858)() at functional:1558:16
> frame #18: 0x0000000105268ac9
> libkudu_client.dylib`std::__1::__function::__func<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
> long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0,
> std::__1::allocator<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
> long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0>, void
> ()>::operator(this=0x0000000112899850)() at functional:1732:12
> frame #19: 0x00000001013ae082
> libtserver_test_util.dylib`std::__1::__function::__value_func<void
> ()>::operator(this=0x0000000112899850)() const at functional:1885:16
> frame #20: 0x00000001013adee5
> libtserver_test_util.dylib`std::__1::function<void
> ()>::operator(this=0x0000000112899850)() const at functional:2560:12
> frame #21: 0x000000010a16cd62
> libkrpc.dylib`kudu::rpc::PeriodicTimer::Callback(this=0x0000000112899830,
> my_callback_generation=1) at periodic.cc:194:5
> frame #22: 0x000000010a17159a
> libkrpc.dylib`kudu::rpc::PeriodicTimer::Callback(this=0x0000000111d62528,
> s=0x000070001a95e910)::$_0::operator()(kudu::Status const&) const at
> periodic.cc:214:14
> frame #23: 0x000000010a171512
> libkrpc.dylib`decltype(__f=0x0000000111d62528,
> __args=0x000070001a95e910)::$_0&>(fp)(std::__1::forward<kudu::Status
> const&>(fp0))) std::__1::__invoke<kudu::rpc::PeriodicTimer::Callback(long
> long)::$_0&, kudu::Status const&>(kudu::rpc::PeriodicTimer::Callback(long
> long)::$_0&, kudu::Status const&) at type_traits:3694:1
> frame #24: 0x000000010a1714b2 libkrpc.dylib`void
> std::__1::__invoke_void_return_wrapper<void,
> true>::__call<kudu::rpc::PeriodicTimer::Callback(__args=0x0000000111d62528,
> __args=0x000070001a95e910)::$_0&, kudu::Status
> const&>(kudu::rpc::PeriodicTimer::Callback(long long)::$_0&, kudu::Status
> const&) at __functional_base:348:9
> frame #25: 0x000000010a171462
> libkrpc.dylib`std::__1::__function::__alloc_func<kudu::rpc::PeriodicTimer::Callback(long
> long)::$_0, std::__1::allocator<kudu::rpc::PeriodicTimer::Callback(long
> long)::$_0>, void (kudu::Status const&)>::operator(this=0x0000000111d62528,
> __arg=0x000070001a95e910)(kudu::Status const&) at functional:1558:16
> frame #26: 0x000000010a16ff11
> libkrpc.dylib`std::__1::__function::__func<kudu::rpc::PeriodicTimer::Callback(long
> long)::$_0, std::__1::allocator<kudu::rpc::PeriodicTimer::Callback(long
> long)::$_0>, void (kudu::Status const&)>::operator(this=0x0000000111d62520,
> __arg=0x000070001a95e910)(kudu::Status const&) at functional:1732:12
> frame #27: 0x0000000101b256da
> libmaster.dylib`std::__1::__function::__value_func<void (kudu::Status
> const&)>::operator(this=0x0000000111d62520,
> __args=0x000070001a95e910)(kudu::Status const&) const at functional:1885:16
> frame #28: 0x0000000101b17cbd libmaster.dylib`std::__1::function<void
> (kudu::Status const&)>::operator(this= Lambda in File periodic.cc at Line
> 208, __arg=0x000070001a95e910)(kudu::Status const&) const at
> functional:2560:12
> frame #29: 0x000000010a188f89
> libkrpc.dylib`kudu::rpc::DelayedTask::TimerHandler(this=0x0000000111d62500,
> (null)=0x0000000111d62560, revents=256) at reactor.cc:767:5
> frame #30: 0x000000010a1965ce libkrpc.dylib`void ev::base<ev_timer,
> ev::timer>::method_thunk<kudu::rpc::DelayedTask, &(loop=0x00000001136f78c0,
> w=0x0000000111d62560, revents=256))>(ev_loop*, ev_timer*, int) at ev++.h:479:7
> frame #31: 0x000000010f327d41
> libev.4.dylib`ev_invoke_pending(loop=0x00000001136f78c0) at ev.c:3155:11
> frame #32: 0x000000010a18027e
> libkrpc.dylib`kudu::rpc::ReactorThread::InvokePendingCb(loop=0x00000001136f78c0)
> at reactor.cc:204:3
> frame #33: 0x000000010f3283ce
> libev.4.dylib`ev_run(loop=0x00000001136f78c0, flags=0) at ev.c:3555:7
> frame #34: 0x000000010a186dbe
> libkrpc.dylib`ev::loop_ref::run(this=0x0000000113502a18, flags=0) at
> ev++.h:211:7
> frame #35: 0x000000010a186ba8
> libkrpc.dylib`kudu::rpc::ReactorThread::RunThread(this=0x0000000113502a10) at
> reactor.cc:505:9
> frame #36: 0x000000010a190fd8
> libkrpc.dylib`kudu::rpc::ReactorThread::Init(this=0x00000001133ab378)::$_0::operator()()
> const at reactor.cc:196:48
> frame #37: 0x000000010a190f9d
> libkrpc.dylib`decltype(__f=0x00000001133ab378)::$_0&>(fp)())
> std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&)
> at type_traits:3694:1
> frame #38: 0x000000010a190f4d libkrpc.dylib`void
> std::__1::__invoke_void_return_wrapper<void,
> true>::__call<kudu::rpc::ReactorThread::Init(__args=0x00000001133ab378)::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&)
> at __functional_base:348:9
> frame #39: 0x000000010a190f1d
> libkrpc.dylib`std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0,
> std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void
> ()>::operator(this=0x00000001133ab378)() at functional:1558:16
> frame #40: 0x000000010a18fab9
> libkrpc.dylib`std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0,
> std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void
> ()>::operator(this=0x00000001133ab370)() at functional:1732:12
> frame #41: 0x00000001013ae082
> libtserver_test_util.dylib`std::__1::__function::__value_func<void
> ()>::operator(this=0x00000001133ab370)() const at functional:1885:16
> frame #42: 0x00000001013adee5
> libtserver_test_util.dylib`std::__1::function<void ()>::operator(this= Lambda
> in File reactor.cc at Line 196)() const at functional:2560:12
> frame #43: 0x000000010e4f92e7
> libkudu_util.dylib`kudu::Thread::SuperviseThread(arg=0x00000001133ab320) at
> thread.cc:691:3
> frame #44: 0x00007fff205e78fc libsystem_pthread.dylib`_pthread_start + 224
> frame #45: 0x00007fff205e3443 libsystem_pthread.dylib`thread_start + 15
> {noformat}
> The version information is below:
> {noformat}
> $ ./bin/client-test --version
> kudu 1.18.0-SNAPSHOT
> revision 8644d88dae6a76c5df3595b8a5aeb13df4d6ab5c
> build type DEBUG
> built by ... at 13 Nov 2023 11:32:09 PST on ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
