Hi Martin,

Could you additionally do a minute worth of MDS perf dump captures at
5 second intervals during the observed slowness? This can be done
using

> ceph tell mds.<id> perf dump

and pipe the result to separate (time-stamped) files and share those.

On Fri, Feb 13, 2026 at 4:59 PM Martin Gerhard Loschwitz via
ceph-users <[email protected]> wrote:
>
> Folks,
>
> we are seeing very slow MDSes on a CephFS that has nearly nothing to do. Slow 
> as in „200 metadata requests per second“ slow when putting a bit of load onto 
> it. We managed to get a wallclock of the MDS:
>
> [root@node01 gdbpmp]# ./gdbpmp.py -i gdbpmp.data
>
> Thread: 7 (mds-rank-fin) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% Finisher::finisher_thread_entry()
>    + 94.12% MDSIOContextBase::complete
>    |+ 88.24% MDSRank::respawn
>    ||+ 88.24% Context::complete
>    || + 88.24% MDSDaemon::respawn
>    ||  + 88.24% ceph::logging::Log::dump_recent()
>    ||   + 52.94% 
> ceph::logging::Log::_flush(std::vector<ceph::logging::ConcreteEntry, 
> std::allocator<ceph::logging::ConcreteEntry> >&, bool)
>    ||   |+ 23.53% __snprintf_chk
>    ||   ||+ 23.53% __vsnprintf_internal
>    ||   || + 17.65% __vfprintf_internal
>    ||   ||  + 5.88% __strchrnul_evex
>    ||   ||  + 5.88% _IO_default_xsputn
>    ||   |+ 17.65% ceph::logging::Log::_flush_logbuf()
>    ||   ||+ 17.65% 
> ceph::logging::Log::_log_safe_write(std::basic_string_view<char, 
> std::char_traits<char> >)
>    ||   || + 17.65% safe_write
>    ||   ||  + 17.65% write
>    ||   |+ 5.88% __tz_convert
>    ||   | + 5.88% __tzfile_compute
>    ||   |  + 5.88% __strcmp_evex
>    ||   + 5.88% __memmove_avx512_unaligned_erms
>    |+ 5.88% std::lock_guard<ceph::fair_mutex>::lock_guard
>    | + 5.88% ceph::fair_mutex::lock
>    |  + 5.88% 
> std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
>  ceph::fair_mutex::lock()::{lambda()#1})
>    |   + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>    |    + 5.88% pthread_cond_wait@@GLIBC_2.3.2
>    |     + 5.88% __futex_abstimed_wait_common
>    + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>     + 5.88% pthread_cond_wait@@GLIBC_2.3.2
>      + 5.88% __futex_abstimed_wait_common
>
> Thread: 10 (mds-rank-progr) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% MDSRank::ProgressThread::entry
>    + 100.00% 
> std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex>, 
> MDSRank::ProgressThread::entry()::<lambda()> >
>     + 100.00% 
> std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex> >
>      + 88.24% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>      |+ 88.24% pthread_cond_wait@@GLIBC_2.3.2
>      | + 88.24% __futex_abstimed_wait_common
>      + 11.76% 
> std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> 
> >::~_Unlock
>       + 11.76% std::unique_lock<ceph::fair_mutex>::lock
>        + 11.76% ceph::fair_mutex::lock
>         + 11.76% 
> std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
>  ceph::fair_mutex::lock()::{lambda()#1})
>          + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>           + 11.76% pthread_cond_wait@@GLIBC_2.3.2
>            + 11.76% __futex_abstimed_wait_common
>
> Thread: 11 (mds-log-trim) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% execute_native_thread_routine
>    + 100.00% MDLog::log_trim_upkeep
>     + 100.00% 
> std::_V2::condition_variable_any::wait_for<std::unique_lock<ceph::fair_mutex>,
>  long, std::ratio<1l, 1000l> >
>      + 100.00% 
> std::_V2::condition_variable_any::wait_until<std::unique_lock<ceph::fair_mutex>,
>  std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 
> 1000000000l> > >
>       + 82.35% 
> std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> 
> >::~_Unlock
>       |+ 82.35% std::unique_lock<ceph::fair_mutex>::lock
>       | + 82.35% ceph::fair_mutex::lock
>       |  + 82.35% 
> std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
>  ceph::fair_mutex::lock()::{lambda()#1})
>       |   + 82.35% 
> std::condition_variable::wait(std::unique_lock<std::mutex>&)
>       |    + 82.35% pthread_cond_wait@@GLIBC_2.3.2
>       |     + 82.35% __futex_abstimed_wait_common
>       + 17.65% 
> std::condition_variable::wait_until<std::chrono::duration<long, 
> std::ratio<1l, 1000000000l> > >
>        + 17.65% 
> std::condition_variable::__wait_until_impl<std::chrono::duration<long, 
> std::ratio<1l, 1000000000l> > >
>         + 17.65% std::__condvar::wait_until
>          + 17.65% pthread_cond_clockwait@GLIBC_2.30
>           + 17.65% __futex_abstimed_wait_common
>
> Thread: 12 (mds-cache-trim) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% execute_native_thread_routine
>    + 100.00% MDCache::upkeep_main
>     + 70.59% std::scoped_lock<ceph::fair_mutex>::scoped_lock
>     |+ 70.59% ceph::fair_mutex::lock
>     | + 70.59% 
> std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
>  ceph::fair_mutex::lock()::{lambda()#1})
>     |  + 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>     |   + 70.59% pthread_cond_wait@@GLIBC_2.3.2
>     |    + 70.59% __futex_abstimed_wait_common
>     + 23.53% std::condition_variable::wait_for<long, std::ratio<1l, 
> 1000000000l> >
>     |+ 23.53% std::condition_variable::wait_until<std::chrono::duration<long, 
> std::ratio<1l, 1000000000l> > >
>     | + 23.53% 
> std::condition_variable::__wait_until_impl<std::chrono::duration<long, 
> std::ratio<1l, 1000000000l> > >
>     |  + 23.53% std::__condvar::wait_until
>     |   + 23.53% pthread_cond_clockwait@GLIBC_2.30
>     |    + 23.53% __futex_abstimed_wait_common
>     + 5.88% MDCache::check_memory_usage
>      + 5.88% MemoryModel::sample
>       + 5.88% MemoryModel::_sample
>        + 5.88% std::getline<char, std::char_traits<char>, 
> std::allocator<char> >
>         + 5.88% std::basic_istream<char, std::char_traits<char> >& 
> std::getline<char, std::char_traits<char>, std::allocator<char> 
> >(std::basic_istream<char, std::char_traits<char> >&, 
> std::__cxx11::basic_string<char
>                 , std::char_traits<char>, std::allocator<char> >&, char)
>          + 5.88% std::basic_filebuf<char, std::char_traits<char> 
> >::underflow()
>           + 5.88% std::__basic_file<char>::xsgetn(char*, long)
>            + 5.88% read
>
>
> Thread: 15 (safe_timer) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% CommonSafeTimerThread<std::mutex>::entry()
>    + 100.00% CommonSafeTimer<std::mutex>::timer_thread()
>     + 88.24% pthread_cond_clockwait@GLIBC_2.30
>     |+ 88.24% __futex_abstimed_wait_common
>     + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>      + 11.76% pthread_cond_wait@@GLIBC_2.3.2
>       + 11.76% __futex_abstimed_wait_common
>
> Thread: 17 (safe_timer) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% CommonSafeTimerThread<ceph::fair_mutex>::entry()
>    + 100.00% CommonSafeTimer<ceph::fair_mutex>::timer_thread()
>     + 70.59% 
> std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> 
> >::~_Unlock()
>     |+ 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>     | + 70.59% pthread_cond_wait@@GLIBC_2.3.2
>     |  + 70.59% __futex_abstimed_wait_common
>     + 29.41% pthread_cond_clockwait@GLIBC_2.30
>      + 29.41% __futex_abstimed_wait_common
>
>
> Thread: 20 (ms_dispatch) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% DispatchQueue::DispatchThread::entry()
>    + 100.00% DispatchQueue::entry()
>     + 100.00% MDSDaemon::ms_dispatch2
>      + 94.12% std::lock_guard<ceph::fair_mutex>::lock_guard
>      |+ 94.12% ceph::fair_mutex::lock
>      | + 94.12% 
> std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
>  ceph::fair_mutex::lock()::{lambda()#1})
>      |  + 94.12% std::condition_variable::wait(std::unique_lock<std::mutex>&)
>      |   + 94.12% pthread_cond_wait@@GLIBC_2.3.2
>      |    + 94.12% __futex_abstimed_wait_common
>      + 5.88% MDSRankDispatcher::ms_dispatch
>       + 5.88% MDSRank::_dispatch
>        + 5.88% MDSRank::_advance_queues
>         + 5.88% MDSContext::complete
>          + 5.88% Context::complete
>           + 5.88% Server::handle_client_readdir
>            + 5.88% CInode::encode_inodestat
>             + 5.88% 
> encode<optmetadata_singleton<optmetadata_client_t<std::allocator>, 
> std::allocator>, std::allocator>
>              + 5.88% 
> optmetadata_multiton<optmetadata_singleton<optmetadata_client_t<std::allocator>,
>  std::allocator>, std::allocator>::encode
>               + 5.88% ceph::buffer::v15_2_0::list::append_hole(unsigned int)
>                + 5.88% 
> ceph::buffer::v15_2_0::list::refill_append_space(unsigned int)
>                 + 5.88% mempool::pool_t::adjust_count(long, long)
>
> Thread: 21 (io_context_pool) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% execute_native_thread_routine
>    + 100.00% 
> std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::i
>              o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, 
> (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> 
> > >::_M_run()
>     + 100.00% 
> std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::io_context_pool::start(sh
>               ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()()
>      + 100.00% 
> std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::io_context_pool::start(s
>                hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 
> 1ul>(std::_Index_tuple<0ul, 1ul>)
>       + 100.00% 
> std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, 
> ceph::async::io_context_pool::start(short)::{lambda()#1
>                 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&,
>  ceph::async::io_context_pool::start(sho
>                 rt)::{lambda()#1}&&)
>        + 100.00% std::__invoke_impl<void, 
> make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::io_context_pool::start(short):
>                  :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
>  make_named_thread<ceph::async::io_context_pool::start(short)::{la
>                  mbda()#1}>(std::basic_string_view<char, 
> std::char_traits<char> >, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&,
>  (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta
>                  rt(short)::{lambda()#1}&&)
>         + 100.00% 
> make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd
>                   a(auto:1&&, 
> (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>  const
>          + 100.00% 
> std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>           + 100.00% 
> std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>            + 100.00% std::__invoke_impl<void, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
>  ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>             + 100.00% 
> ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const
>              + 100.00% boost::asio::io_context::run
>               + 100.00% 
> boost::asio::detail::scheduler::run(boost::system::error_code&) [clone 
> .constprop.0] [clone .isra.0]
>                + 100.00% boost::asio::detail::scheduler::do_run_one
>                 + 94.12% 
> boost::asio::detail::conditionally_enabled_event::wait
>                 |+ 94.12% 
> boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock>
>                 | + 94.12% pthread_cond_wait@@GLIBC_2.3.2
>                 |  + 94.12% __futex_abstimed_wait_common
>                 + 5.88% boost::asio::detail::scheduler_operation::complete
>                  + 5.88% 
> boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
>  0ul> const, void>, std::allocator<voi
>                          d>, 
> boost::asio::detail::scheduler_operation>::do_complete(void*, 
> boost::asio::detail::scheduler_operation*, boost::system::error_code const&, 
> unsigned long)
>                   + 5.88% 
> boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
>  0ul> const, void>::operator()()
>                    + 5.88% 
> boost::asio::detail::executor_op<boost::asio::detail::binder0<MonClient::handle_config(MConfig*)::{lambda()#2}>,
>  std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete
>                            (void*, boost::asio::detail::scheduler_operation*, 
> boost::system::error_code const&, unsigned long) [clone .lto_priv.0]
>                     + 5.88% 
> ceph::common::ConfigProxy::set_mon_vals(ceph::common::CephContext*, 
> std::map<std::__cxx11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> >, std::__cxx11::basic_string<cha
>                             r, std::char_traits<char>, std::allocator<char> 
> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> > const, std::__c
>                             xx11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > > > > const&, std::function<bool 
> (std::__cxx11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > co
>                             nst&, std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> > const&)>) [clone .isra.0]
>                      + 5.88% 
> md_config_t::set_mon_vals(ceph::common::CephContext*, ConfigValues&, 
> ConfigTracker const&, std::map<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> >, std::_
>                              _cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> >, std::less<void>, 
> std::allocator<std::pair<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocato
>                              r<char> > const, 
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> 
> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, 
> std::char_traits<char>,
>                               std::allocator<char> > const&, 
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> 
> > const&)>)
>                       + 5.88% md_config_t::update_legacy_vals(ConfigValues&)
>                        + 5.88% md_config_t::update_legacy_val(ConfigValues&, 
> Option const&, std::variant<long ConfigValues::*, unsigned long 
> ConfigValues::*, std::__cxx11::basic_string<char, std::char_traits<char>, s
>                                td::allocator<char> > ConfigValues::*, double 
> ConfigValues::*, bool ConfigValues::*, entity_addr_t ConfigValues::*, 
> entity_addrvec_t ConfigValues::*, uuid_d ConfigValues::*>)
>                         + 5.88% md_config_t::_get_val(ConfigValues const&, 
> Option const&, boost::container::small_vector<std::pair<Option const*, 
> std::variant<std::monostate, std::__cxx11::basic_string<char, std::cha
>                                 r_traits<char>, std::allocator<char> >, 
> unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, 
> std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long
>                                 , std::ratio<1l, 1000l> >, Option::size_t, 
> uuid_d> const*>, 4ul, void, void>*, std::basic_ostream<char, 
> std::char_traits<char> >*) const
>                          + 5.88% 
> md_config_t::_get_val_nometa[abi:cxx11](ConfigValues const&, Option const&) 
> const
>
> Thread: 22 (io_context_pool) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% execute_native_thread_routine
>    + 100.00% 
> std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::i
>              o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, 
> (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> 
> > >::_M_run()
>     + 100.00% 
> std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::io_context_pool::start(sh
>               ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()()
>      + 100.00% 
> std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::io_context_pool::start(s
>                hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 
> 1ul>(std::_Index_tuple<0ul, 1ul>)
>       + 100.00% 
> std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, 
> ceph::async::io_context_pool::start(short)::{lambda()#1
>                 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&,
>  ceph::async::io_context_pool::start(sho
>                 rt)::{lambda()#1}&&)
>        + 100.00% std::__invoke_impl<void, 
> make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, ceph::async::io_context_pool::start(short):
>                  :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
>  make_named_thread<ceph::async::io_context_pool::start(short)::{la
>                  mbda()#1}>(std::basic_string_view<char, 
> std::char_traits<char> >, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&,
>  (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta
>                  rt(short)::{lambda()#1}&&)
>         + 100.00% 
> make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
>  std::char_traits<char> >, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd
>                   a(auto:1&&, 
> (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>  const
>          + 100.00% 
> std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>           + 100.00% 
> std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>            + 100.00% std::__invoke_impl<void, 
> ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
>  ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
>             + 100.00% 
> ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const
>              + 100.00% boost::asio::io_context::run
>               + 100.00% 
> boost::asio::detail::scheduler::run(boost::system::error_code&) [clone 
> .constprop.0] [clone .isra.0]
>                + 100.00% boost::asio::detail::scheduler::do_run_one
>                 + 100.00% 
> boost::asio::detail::conditionally_enabled_event::wait
>                  + 100.00% 
> boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock>
>                   + 100.00% pthread_cond_wait@@GLIBC_2.3.2
>                    + 100.00% __futex_abstimed_wait_common
>
> Thread: 26 (msgr-worker-2) - 17 samples
>
> + 100.00% clone
>  + 100.00% start_thread
>   + 100.00% execute_native_thread_routine
>    + 100.00% std::_Function_handler<void (), 
> NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data 
> const&)
>     + 100.00% EventCenter::process_events(unsigned int, 
> std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
>      + 94.12% EpollDriver::event_wait(std::vector<FiredFileEvent, 
> std::allocator<FiredFileEvent> >&, timeval*)
>      |+ 94.12% epoll_wait
>      + 5.88% AsyncConnection::process()
>       + 5.88% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
>        + 5.88% ProtocolV2::handle_message()
>         + 5.88% decode_message(ceph::common::CephContext*, int, 
> ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v15_2_0::list&, 
> ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, 
> boost::intrusive_ptr<Conne
>                 ction>)
>          + 5.88% MMgrMap::decode_payload()
>           + 5.88% 
> MgrMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)
>            + 5.88% 
> MgrMap::ModuleInfo::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)
>             + 5.88% 
> std::enable_if<denc_traits<std::set<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> >, 
> std::less<std::__cxx11::basic_string<char, std::char_traits<char>, 
> std::allocat
>                     or<char> > >, 
> std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > > >, 
> void>::supported&&(!denc_traits<std::set<std::__cxx11::basic_string<char, 
> std::cha
>                     r_traits<char>, std::allocator<char> >, 
> std::less<std::__cxx11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, 
> std::char_trait
>                     s<char>, std::allocator<char> > > >, 
> void>::need_contiguous), void>::type 
> ceph::decode<std::set<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> >, std::less<std::__cx
>                     x11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> > > >, denc_traits<std::set<s
>                     td::__cxx11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> >, std::less<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__c
>                     xx11::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > > >, void> >(std::set<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx1
>                     1::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, 
> std::char_traits<char>, std::allocator<char> > > >&, ceph::buffer::v15_2_0::
>                     list::iterator_impl<true>&)
>              + 5.88% ceph::buffer::v15_2_0::ptr::release()
>
> Sidenote: while recording this, the MDS became so slow that the MDS cluster 
> declared it dead … As can be seen, the MDS is waiting in futex to no end, but 
> how can we find out what it is waiting for? FS has many small files but 
> nearly no activity ….
>
> Thanks in advance
> Martin
>
> --
> Martin Gerhard Loschwitz
> Geschäftsführer / CEO, True West IT Services GmbH
> Phone: +49 2433 5253130
> Mobile: +49 176 61832178
> Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany
> Legal: HRB 21985, Amtsgericht Mönchengladbach
> VAT: DE363893844
>
> True West IT Services GmbH is compliant with the GDPR regulation on data 
> protection and privacy in the European Union and the European Economic Area. 
> You can request the information on how we collect and process your private 
> data according to the law by contacting the email sender.
>
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]



-- 
Cheers,
Venky
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]

Reply via email to