On Wed, Jul 25, 2018 at 5:04 PM Daniel Carrasco <d.carra...@i2tic.com> wrote: > > Hello, > > I've attached the PDF. > > I don't know if is important, but I made changes on configuration and I've > restarted the servers after dump that heap file. I've changed the > memory_limit to 25Mb to test if stil with aceptable values of RAM. >
Looks like there are memory leak in async messenger. what's output of "dd /usr/bin/ceph-mds"? Could you try simple messenger (add "ms type = simple" to 'global' section of ceph.conf) Regards Yan, Zheng > Greetings! > > 2018-07-25 2:53 GMT+02:00 Yan, Zheng <uker...@gmail.com>: >> >> On Wed, Jul 25, 2018 at 4:52 AM Daniel Carrasco <d.carra...@i2tic.com> wrote: >> > >> > Hello, >> > >> > I've run the profiler for about 5-6 minutes and this is what I've got: >> > >> >> please run pprof --pdf /usr/bin/ceph-mds >> /var/log/ceph/ceph-mds.x.profile.<largest number>.heap > >> /tmp/profile.pdf. and send me the pdf >> >> >> >> > -------------------------------------------------------------------------------------------- >> > -------------------------------------------------------------------------------------------- >> > -------------------------------------------------------------------------------------------- >> > Using local file /usr/bin/ceph-mds. >> > Using local file >> > /var/log/ceph/mds.kavehome-mgto-pro-fs01.profile.0009.heap. >> > Total: 400.0 MB >> > 362.5 90.6% 90.6% 362.5 90.6% >> > ceph::buffer::create_aligned_in_mempool >> > 20.4 5.1% 95.7% 29.8 7.5% CDir::_load_dentry >> > 5.9 1.5% 97.2% 6.9 1.7% CDir::add_primary_dentry >> > 4.7 1.2% 98.4% 4.7 1.2% ceph::logging::Log::create_entry >> > 1.8 0.5% 98.8% 1.8 0.5% >> > std::_Rb_tree::_M_emplace_hint_unique >> > 1.8 0.5% 99.3% 2.2 0.5% compact_map_base::decode >> > 0.6 0.1% 99.4% 0.7 0.2% CInode::add_client_cap >> > 0.5 0.1% 99.5% 0.5 0.1% >> > std::__cxx11::basic_string::_M_mutate >> > 0.4 0.1% 99.6% 0.4 0.1% SimpleLock::more >> > 0.4 0.1% 99.7% 0.4 0.1% MDCache::add_inode >> > 0.3 0.1% 99.8% 0.3 0.1% CDir::add_to_bloom >> > 0.2 0.1% 99.9% 0.2 0.1% CDir::steal_dentry >> > 0.2 0.0% 99.9% 0.2 0.0% CInode::get_or_open_dirfrag >> > 0.1 0.0% 99.9% 0.8 0.2% std::enable_if::type decode >> > 0.1 0.0% 100.0% 0.1 0.0% ceph::buffer::list::crc32c >> > 0.1 0.0% 100.0% 0.1 0.0% decode_message >> > 0.0 0.0% 100.0% 0.0 0.0% OpTracker::create_request >> > 0.0 0.0% 100.0% 0.0 0.0% TrackedOp::TrackedOp >> > 0.0 0.0% 100.0% 0.0 0.0% std::vector::_M_emplace_back_aux >> > 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_unique >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::add_dirfrag >> > 0.0 0.0% 100.0% 0.0 0.0% MDLog::_prepare_new_segment >> > 0.0 0.0% 100.0% 0.0 0.0% DispatchQueue::enqueue >> > 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::push_back >> > 0.0 0.0% 100.0% 0.0 0.0% Server::prepare_new_inode >> > 0.0 0.0% 100.0% 365.6 91.4% EventCenter::process_events >> > 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_copy >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::add_null_dentry >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::check_inode_max_size >> > 0.0 0.0% 100.0% 0.0 0.0% CDentry::add_client_lease >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::project_inode >> > 0.0 0.0% 100.0% 0.0 0.0% std::__cxx11::list::_M_insert >> > 0.0 0.0% 100.0% 0.0 0.0% MDBalancer::handle_heartbeat >> > 0.0 0.0% 100.0% 0.0 0.0% MDBalancer::send_heartbeat >> > 0.0 0.0% 100.0% 0.0 0.0% C_GatherBase::C_GatherSub::complete >> > 0.0 0.0% 100.0% 0.0 0.0% EventCenter::create_time_event >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::_omap_fetch >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::handle_inode_file_caps >> > 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_equal >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::issue_caps >> > 0.0 0.0% 100.0% 0.1 0.0% MDLog::_submit_thread >> > 0.0 0.0% 100.0% 0.0 0.0% Journaler::_wait_for_flush >> > 0.0 0.0% 100.0% 0.0 0.0% Journaler::wrap_finisher >> > 0.0 0.0% 100.0% 0.0 0.0% MDSCacheObject::add_waiter >> > 0.0 0.0% 100.0% 0.0 0.0% std::__cxx11::list::insert >> > 0.0 0.0% 100.0% 0.0 0.0% std::__detail::_Map_base::operator[] >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::mark_updated_scatterlock >> > 0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_ >> > 0.0 0.0% 100.0% 0.0 0.0% alloc_ptr::operator-> >> > 0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::append@5c1560 >> > 0.0 0.0% 100.0% 0.0 0.0% >> > ceph::buffer::malformed_input::~malformed_input >> > 0.0 0.0% 100.0% 0.0 0.0% compact_set_base::insert >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::add_waiter >> > 0.0 0.0% 100.0% 0.0 0.0% InoTable::apply_release_ids >> > 0.0 0.0% 100.0% 0.0 0.0% InoTable::project_release_ids >> > 0.0 0.0% 100.0% 2.2 0.5% InodeStoreBase::decode_bare >> > 0.0 0.0% 100.0% 0.0 0.0% interval_set::erase >> > 0.0 0.0% 100.0% 1.1 0.3% std::map::operator[] >> > 0.0 0.0% 100.0% 0.0 0.0% Beacon::_send >> > 0.0 0.0% 100.0% 0.0 0.0% MDSDaemon::reset_tick >> > 0.0 0.0% 100.0% 0.0 0.0% MgrClient::send_report >> > 0.0 0.0% 100.0% 0.0 0.0% Journaler::_do_flush >> > 0.0 0.0% 100.0% 0.1 0.0% Locker::rdlock_start >> > 0.0 0.0% 100.0% 0.0 0.0% MDCache::_get_waiter >> > 0.0 0.0% 100.0% 0.0 0.0% CDentry::~CDentry >> > 0.0 0.0% 100.0% 0.0 0.0% MonClient::schedule_tick >> > 0.0 0.0% 100.0% 0.1 0.0% AsyncConnection::handle_write >> > 0.0 0.0% 100.0% 0.1 0.0% >> > AsyncConnection::prepare_send_message >> > 0.0 0.0% 100.0% 365.5 91.4% AsyncConnection::process >> > 0.0 0.0% 100.0% 0.3 0.1% AsyncConnection::send_message >> > 0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::tick >> > 0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::_send_message >> > 0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::send_message >> > 0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::submit_message >> > 0.0 0.0% 100.0% 0.0 0.0% Beacon::notify_health >> > 0.0 0.0% 100.0% 0.2 0.1% CDentry::CDentry >> > 0.0 0.0% 100.0% 0.0 0.0% CDentry::_mark_dirty >> > 0.0 0.0% 100.0% 0.0 0.0% CDentry::auth_pin >> > 0.0 0.0% 100.0% 0.0 0.0% CDentry::mark_dirty >> > 0.0 0.0% 100.0% 0.0 0.0% CDentry::pop_projected_linkage >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::_mark_dirty >> > 0.0 0.0% 100.0% 29.8 7.5% CDir::_omap_fetched >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::auth_pin >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::fetch >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::link_inode_work >> > 0.0 0.0% 100.0% 0.0 0.0% CDir::link_primary_inode >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::_mark_dirty >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::add_waiter >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::auth_pin >> > 0.0 0.0% 100.0% 0.4 0.1% CInode::choose_ideal_loner >> > 0.0 0.0% 100.0% 0.4 0.1% CInode::encode_inodestat >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::mark_dirty >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::mark_dirty_parent >> > 0.0 0.0% 100.0% 0.0 0.0% CInode::mark_dirty_rstat >> > 0.0 0.0% 100.0% 0.0 0.0% >> > CInode::pop_and_dirty_projected_inode >> > 0.0 0.0% 100.0% 0.4 0.1% CInode::set_loner_cap >> > 0.0 0.0% 100.0% 29.8 7.5% C_IO_Dir_OMAP_Fetched::finish >> > 0.0 0.0% 100.0% 0.0 0.0% C_Locker_FileUpdate_finish::finish >> > 0.0 0.0% 100.0% 0.0 0.0% C_MDL_CheckMaxSize::finish >> > 0.0 0.0% 100.0% 0.0 0.0% >> > C_MDS_RetryRequest::~C_MDS_RetryRequest >> > 0.0 0.0% 100.0% 0.0 0.0% C_MDS_inode_update_finish::finish >> > 0.0 0.0% 100.0% 0.0 0.0% C_MDS_openc_finish::finish >> > 0.0 0.0% 100.0% 0.0 0.0% C_MDS_session_finish::finish >> > 0.0 0.0% 100.0% 0.0 0.0% C_OnFinisher::finish >> > 0.0 0.0% 100.0% 1.2 0.3% Context::complete >> > 0.0 0.0% 100.0% 3.7 0.9% DispatchQueue::DispatchThread::entry >> > 0.0 0.0% 100.0% 3.7 0.9% DispatchQueue::entry >> > 0.0 0.0% 100.0% 0.9 0.2% DispatchQueue::fast_dispatch >> > 0.0 0.0% 100.0% 2.1 0.5% DispatchQueue::pre_dispatch >> > 0.0 0.0% 100.0% 0.0 0.0% EMetaBlob::print >> > 0.0 0.0% 100.0% 0.0 0.0% EventCenter::process_time_events >> > 0.0 0.0% 100.0% 29.9 7.5% Finisher::finisher_thread_entry >> > 0.0 0.0% 100.0% 0.4 0.1% FunctionContext::finish >> > 0.0 0.0% 100.0% 0.0 0.0% Journaler::_flush >> > 0.0 0.0% 100.0% 0.0 0.0% Journaler::_write_head >> > 0.0 0.0% 100.0% 0.0 0.0% Journaler::flush >> > 0.0 0.0% 100.0% 0.0 0.0% Journaler::wait_for_flush >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::_do_cap_release >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::_do_cap_update >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::_drop_non_rdlocks >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::_rdlock_kick >> > 0.0 0.0% 100.0% 0.2 0.0% Locker::acquire_locks >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::adjust_cap_wanted >> > 0.0 0.0% 100.0% 0.2 0.0% Locker::dispatch >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::drop_locks >> > 0.0 0.0% 100.0% 0.4 0.1% Locker::eval >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::eval_gather >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::file_update_finish >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::handle_client_cap_release >> > 0.0 0.0% 100.0% 0.2 0.0% Locker::handle_client_caps >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::handle_client_lease >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::handle_file_lock >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::handle_lock >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::issue_caps_set >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::issue_client_lease >> > 0.0 0.0% 100.0% 0.6 0.2% Locker::issue_new_caps >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::local_wrlock_start >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::nudge_log >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_eval >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_mix >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_nudge >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_tick >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_writebehind >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_writebehind_finish >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::send_lock_message@42d5b0 >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::send_lock_message@42f2b0 >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::share_inode_max_size >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::simple_lock >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::simple_sync >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::tick >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::try_eval@43da60 >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::try_eval@441fb0 >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::wrlock_finish >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::wrlock_force >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::wrlock_start >> > 0.0 0.0% 100.0% 0.0 0.0% Locker::xlock_start >> > 0.0 0.0% 100.0% 0.0 0.0% MClientCaps::print >> > 0.0 0.0% 100.0% 0.0 0.0% MClientRequest::decode_payload >> > 0.0 0.0% 100.0% 0.0 0.0% MClientRequest::print >> > 0.0 0.0% 100.0% 0.0 0.0% MDBalancer::prep_rebalance >> > 0.0 0.0% 100.0% 0.0 0.0% MDBalancer::proc_message >> > 0.0 0.0% 100.0% 0.0 0.0% MDCache::check_memory_usage >> > 0.0 0.0% 100.0% 0.2 0.1% MDCache::path_traverse >> > 0.0 0.0% 100.0% 0.0 0.0% MDCache::predirty_journal_parents >> > 0.0 0.0% 100.0% 0.0 0.0% MDCache::request_cleanup >> > 0.0 0.0% 100.0% 0.0 0.0% MDCache::request_finish >> > 0.0 0.0% 100.0% 0.0 0.0% MDCache::request_start >> > 0.0 0.0% 100.0% 0.3 0.1% MDCache::trim >> > 0.0 0.0% 100.0% 0.3 0.1% MDCache::trim_dentry >> > 0.0 0.0% 100.0% 0.3 0.1% MDCache::trim_lru >> > 0.0 0.0% 100.0% 0.0 0.0% MDCache::truncate_inode >> > 0.0 0.0% 100.0% 0.1 0.0% MDLog::SubmitThread::entry >> > 0.0 0.0% 100.0% 0.0 0.0% MDLog::_start_new_segment >> > 0.0 0.0% 100.0% 0.0 0.0% MDLog::_submit_entry >> > 0.0 0.0% 100.0% 0.0 0.0% MDLog::submit_entry >> > 0.0 0.0% 100.0% 0.0 0.0% MDSCacheObject::finish_waiting >> > 0.0 0.0% 100.0% 0.2 0.1% MDSCacheObject::get >> > 0.0 0.0% 100.0% 1.7 0.4% MDSDaemon::ms_dispatch >> > 0.0 0.0% 100.0% 0.0 0.0% MDSDaemon::tick >> > 0.0 0.0% 100.0% 29.9 7.5% MDSIOContextBase::complete >> > 0.0 0.0% 100.0% 0.7 0.2% MDSInternalContextBase::complete >> > 0.0 0.0% 100.0% 0.0 0.0% MDSLogContextBase::complete >> > 0.0 0.0% 100.0% 0.3 0.1% MDSRank::ProgressThread::entry >> > 0.0 0.0% 100.0% 0.7 0.2% MDSRank::_advance_queues >> > 0.0 0.0% 100.0% 1.7 0.4% MDSRank::_dispatch >> > 0.0 0.0% 100.0% 1.3 0.3% MDSRank::handle_deferrable_message >> > 0.0 0.0% 100.0% 0.0 0.0% MDSRank::send_message_client >> > 0.0 0.0% 100.0% 0.0 0.0% >> > MDSRank::send_message_client_counted@2a9260 >> > 0.0 0.0% 100.0% 0.0 0.0% >> > MDSRank::send_message_client_counted@2a94f0 >> > 0.0 0.0% 100.0% 0.0 0.0% >> > MDSRank::send_message_client_counted@2b1920 >> > 0.0 0.0% 100.0% 0.0 0.0% MDSRank::send_message_mds >> > 0.0 0.0% 100.0% 1.7 0.4% MDSRankDispatcher::ms_dispatch >> > 0.0 0.0% 100.0% 0.4 0.1% MDSRankDispatcher::tick >> > 0.0 0.0% 100.0% 0.0 0.0% MOSDOp::print >> > 0.0 0.0% 100.0% 0.1 0.0% Message::encode >> > 0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_rotating >> > 0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_tickets >> > 0.0 0.0% 100.0% 0.0 0.0% MonClient::_send_mon_message >> > 0.0 0.0% 100.0% 0.0 0.0% MonClient::tick >> > 0.0 0.0% 100.0% 0.0 0.0% MutationImpl::MutationImpl >> > 0.0 0.0% 100.0% 0.1 0.0% MutationImpl::auth_pin >> > 0.0 0.0% 100.0% 0.1 0.0% MutationImpl::pin >> > 0.0 0.0% 100.0% 0.0 0.0% MutationImpl::start_locking >> > 0.0 0.0% 100.0% 365.6 91.4% NetworkStack::get_worker >> > 0.0 0.0% 100.0% 0.8 0.2% >> > ObjectOperation::C_ObjectOperation_decodevals::finish >> > 0.0 0.0% 100.0% 0.1 0.0% Objecter::_op_submit >> > 0.0 0.0% 100.0% 0.1 0.0% Objecter::_op_submit_with_budget >> > 0.0 0.0% 100.0% 0.1 0.0% Objecter::_send_op >> > 0.0 0.0% 100.0% 0.8 0.2% Objecter::handle_osd_op_reply >> > 0.0 0.0% 100.0% 0.8 0.2% Objecter::ms_dispatch >> > 0.0 0.0% 100.0% 0.8 0.2% Objecter::ms_fast_dispatch >> > 0.0 0.0% 100.0% 0.1 0.0% Objecter::op_submit >> > 0.0 0.0% 100.0% 0.0 0.0% Objecter::sg_write_trunc >> > 0.0 0.0% 100.0% 0.0 0.0% OpHistory::insert >> > 0.0 0.0% 100.0% 0.0 0.0% OpTracker::unregister_inflight_op >> > 0.0 0.0% 100.0% 0.0 0.0% PrebufferedStreambuf::overflow >> > 0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_after >> > 0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_at >> > 0.0 0.0% 100.0% 0.4 0.1% SafeTimer::timer_thread >> > 0.0 0.0% 100.0% 0.4 0.1% SafeTimerThread::entry >> > 0.0 0.0% 100.0% 0.0 0.0% Server::_session_logged >> > 0.0 0.0% 100.0% 0.0 0.0% Server::apply_allocated_inos >> > 0.0 0.0% 100.0% 1.1 0.3% Server::dispatch >> > 0.0 0.0% 100.0% 1.7 0.4% Server::dispatch_client_request >> > 0.0 0.0% 100.0% 0.7 0.2% Server::handle_client_getattr >> > 0.0 0.0% 100.0% 0.9 0.2% Server::handle_client_open >> > 0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_openc >> > 0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_readdir >> > 0.0 0.0% 100.0% 1.1 0.3% Server::handle_client_request >> > 0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_session >> > 0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_setattr >> > 0.0 0.0% 100.0% 0.0 0.0% Server::journal_and_reply >> > 0.0 0.0% 100.0% 0.0 0.0% Server::journal_close_session >> > 0.0 0.0% 100.0% 0.3 0.1% Server::rdlock_path_pin_ref >> > 0.0 0.0% 100.0% 0.0 0.0% Server::recall_client_state >> > 0.0 0.0% 100.0% 0.5 0.1% Server::reply_client_request >> > 0.0 0.0% 100.0% 0.5 0.1% Server::respond_to_request >> > 0.0 0.0% 100.0% 0.4 0.1% Server::set_trace_dist >> > 0.0 0.0% 100.0% 0.0 0.0% SessionMap::_mark_dirty >> > 0.0 0.0% 100.0% 0.0 0.0% SessionMap::mark_dirty >> > 0.0 0.0% 100.0% 0.0 0.0% SessionMap::remove_session >> > 0.0 0.0% 100.0% 0.0 0.0% >> > ceph::buffer::list::iterator_impl::copy >> > 0.0 0.0% 100.0% 0.0 0.0% >> > ceph::buffer::list::iterator_impl::copy_shallow >> > 0.0 0.0% 100.0% 400.0 100.0% clone >> > 0.0 0.0% 100.0% 0.0 0.0% filepath::parse_bits >> > 0.0 0.0% 100.0% 0.0 0.0% inode_t::operator= >> > 0.0 0.0% 100.0% 0.0 0.0% operator<<@2a2890 >> > 0.0 0.0% 100.0% 0.0 0.0% operator<<@2c9760 >> > 0.0 0.0% 100.0% 0.0 0.0% operator<<@3eadf0 >> > 0.0 0.0% 100.0% 400.0 100.0% start_thread >> > 0.0 0.0% 100.0% 0.1 0.0% >> > std::__cxx11::basic_string::_M_append >> > 0.0 0.0% 100.0% 0.0 0.0% >> > std::__cxx11::basic_string::_M_replace_aux >> > 0.0 0.0% 100.0% 0.0 0.0% std::__cxx11::list::operator= >> > 0.0 0.0% 100.0% 0.0 0.0% std::__ostream_insert >> > 0.0 0.0% 100.0% 0.0 0.0% std::basic_streambuf::xsputn >> > 0.0 0.0% 100.0% 0.0 0.0% std::num_put::_M_insert_int >> > 0.0 0.0% 100.0% 0.0 0.0% std::num_put::do_put >> > 0.0 0.0% 100.0% 0.0 0.0% std::operator<< >> > 0.0 0.0% 100.0% 0.0 0.0% std::ostream::_M_insert >> > 0.0 0.0% 100.0% 365.6 91.4% std::this_thread::__sleep_for >> > 0.0 0.0% 100.0% 0.0 0.0% utime_t::localtime >> > 0.0 0.0% 100.0% 0.1 0.0% void finish_contexts@2a30f0 >> > -------------------------------------------------------------------------------------------- >> > -------------------------------------------------------------------------------------------- >> > -------------------------------------------------------------------------------------------- >> > >> > >> > Greetings!! >> > >> > 2018-07-24 12:07 GMT+02:00 Yan, Zheng <uker...@gmail.com>: >> >> >> >> On Tue, Jul 24, 2018 at 4:59 PM Daniel Carrasco <d.carra...@i2tic.com> >> >> wrote: >> >> > >> >> > Hello, >> >> > >> >> > How many time is neccesary?, because is a production environment and >> >> > memory profiler + low cache size because the problem, gives a lot of >> >> > CPU usage from OSD and MDS that makes it fails while profiler is >> >> > running. Is there any problem if is done in a low traffic time? (less >> >> > usage and maybe it don't fails, but maybe less info about usage). >> >> > >> >> >> >> just one time, wait a few minutes between start_profiler and >> >> stop_profiler >> >> >> >> > Greetings! >> >> > >> >> > 2018-07-24 10:21 GMT+02:00 Yan, Zheng <uker...@gmail.com>: >> >> >> >> >> >> I mean: >> >> >> >> >> >> ceph tell mds.x heap start_profiler >> >> >> >> >> >> ... wait for some time >> >> >> >> >> >> ceph tell mds.x heap stop_profiler >> >> >> >> >> >> pprof --text /usr/bin/ceph-mds >> >> >> /var/log/ceph/ceph-mds.x.profile.<largest number>.heap >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Jul 24, 2018 at 3:18 PM Daniel Carrasco <d.carra...@i2tic.com> >> >> >> wrote: >> >> >> > >> >> >> > This is what i get: >> >> >> > >> >> >> > -------------------------------------------------------- >> >> >> > -------------------------------------------------------- >> >> >> > -------------------------------------------------------- >> >> >> > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap dump >> >> >> > 2018-07-24 09:05:19.350720 7fc562ffd700 0 client.1452545 >> >> >> > ms_handle_reset on 10.22.0.168:6800/1685786126 >> >> >> > 2018-07-24 09:05:29.103903 7fc563fff700 0 client.1452548 >> >> >> > ms_handle_reset on 10.22.0.168:6800/1685786126 >> >> >> > mds.kavehome-mgto-pro-fs01 dumping heap profile now. >> >> >> > ------------------------------------------------ >> >> >> > MALLOC: 760199640 ( 725.0 MiB) Bytes in use by application >> >> >> > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist >> >> >> > MALLOC: + 246962320 ( 235.5 MiB) Bytes in central cache freelist >> >> >> > MALLOC: + 43933664 ( 41.9 MiB) Bytes in transfer cache freelist >> >> >> > MALLOC: + 41012664 ( 39.1 MiB) Bytes in thread cache freelists >> >> >> > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata >> >> >> > MALLOC: ------------ >> >> >> > MALLOC: = 1102295200 ( 1051.2 MiB) Actual memory used (physical + >> >> >> > swap) >> >> >> > MALLOC: + 4268335104 ( 4070.6 MiB) Bytes released to OS (aka >> >> >> > unmapped) >> >> >> > MALLOC: ------------ >> >> >> > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used >> >> >> > MALLOC: >> >> >> > MALLOC: 33027 Spans in use >> >> >> > MALLOC: 19 Thread heaps in use >> >> >> > MALLOC: 8192 Tcmalloc page size >> >> >> > ------------------------------------------------ >> >> >> > Call ReleaseFreeMemory() to release freelist memory to the OS (via >> >> >> > madvise()). >> >> >> > Bytes released to the OS take up virtual address space but no >> >> >> > physical memory. >> >> >> > >> >> >> > >> >> >> > -------------------------------------------------------- >> >> >> > -------------------------------------------------------- >> >> >> > -------------------------------------------------------- >> >> >> > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats >> >> >> > 2018-07-24 09:14:25.747706 7f94fffff700 0 client.1452578 >> >> >> > ms_handle_reset on 10.22.0.168:6800/1685786126 >> >> >> > 2018-07-24 09:14:25.754034 7f95057fa700 0 client.1452581 >> >> >> > ms_handle_reset on 10.22.0.168:6800/1685786126 >> >> >> > mds.kavehome-mgto-pro-fs01 tcmalloc heap >> >> >> > stats:------------------------------------------------ >> >> >> > MALLOC: 960649328 ( 916.1 MiB) Bytes in use by application >> >> >> > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist >> >> >> > MALLOC: + 108867288 ( 103.8 MiB) Bytes in central cache freelist >> >> >> > MALLOC: + 37179424 ( 35.5 MiB) Bytes in transfer cache freelist >> >> >> > MALLOC: + 40143000 ( 38.3 MiB) Bytes in thread cache freelists >> >> >> > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata >> >> >> > MALLOC: ------------ >> >> >> > MALLOC: = 1157025952 ( 1103.4 MiB) Actual memory used (physical + >> >> >> > swap) >> >> >> > MALLOC: + 4213604352 ( 4018.4 MiB) Bytes released to OS (aka >> >> >> > unmapped) >> >> >> > MALLOC: ------------ >> >> >> > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used >> >> >> > MALLOC: >> >> >> > MALLOC: 33028 Spans in use >> >> >> > MALLOC: 19 Thread heaps in use >> >> >> > MALLOC: 8192 Tcmalloc page size >> >> >> > ------------------------------------------------ >> >> >> > Call ReleaseFreeMemory() to release freelist memory to the OS (via >> >> >> > madvise()). >> >> >> > Bytes released to the OS take up virtual address space but no >> >> >> > physical memory. >> >> >> > >> >> >> > -------------------------------------------------------- >> >> >> > -------------------------------------------------------- >> >> >> > -------------------------------------------------------- >> >> >> > After heap release: >> >> >> > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats >> >> >> > 2018-07-24 09:15:28.540203 7f2f7affd700 0 client.1443339 >> >> >> > ms_handle_reset on 10.22.0.168:6800/1685786126 >> >> >> > 2018-07-24 09:15:28.547153 7f2f7bfff700 0 client.1443342 >> >> >> > ms_handle_reset on 10.22.0.168:6800/1685786126 >> >> >> > mds.kavehome-mgto-pro-fs01 tcmalloc heap >> >> >> > stats:------------------------------------------------ >> >> >> > MALLOC: 710315776 ( 677.4 MiB) Bytes in use by application >> >> >> > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist >> >> >> > MALLOC: + 246471880 ( 235.1 MiB) Bytes in central cache freelist >> >> >> > MALLOC: + 40802848 ( 38.9 MiB) Bytes in transfer cache freelist >> >> >> > MALLOC: + 38689304 ( 36.9 MiB) Bytes in thread cache freelists >> >> >> > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata >> >> >> > MALLOC: ------------ >> >> >> > MALLOC: = 1046466720 ( 998.0 MiB) Actual memory used (physical + >> >> >> > swap) >> >> >> > MALLOC: + 4324163584 ( 4123.8 MiB) Bytes released to OS (aka >> >> >> > unmapped) >> >> >> > MALLOC: ------------ >> >> >> > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used >> >> >> > MALLOC: >> >> >> > MALLOC: 33177 Spans in use >> >> >> > MALLOC: 19 Thread heaps in use >> >> >> > MALLOC: 8192 Tcmalloc page size >> >> >> > ------------------------------------------------ >> >> >> > Call ReleaseFreeMemory() to release freelist memory to the OS (via >> >> >> > madvise()). >> >> >> > Bytes released to the OS take up virtual address space but no >> >> >> > physical memory. >> >> >> > >> >> >> > >> >> >> > The other commands fails with a curl error: >> >> >> > Failed to get profile: curl 'http:///pprof/profile?seconds=30' > >> >> >> > /root/pprof/.tmp.ceph-mds.1532416424.: >> >> >> > >> >> >> > >> >> >> > Greetings!! >> >> >> > >> >> >> > 2018-07-24 5:35 GMT+02:00 Yan, Zheng <uker...@gmail.com>: >> >> >> >> >> >> >> >> could you profile memory allocation of mds >> >> >> >> >> >> >> >> http://docs.ceph.com/docs/mimic/rados/troubleshooting/memory-profiling/ >> >> >> >> On Tue, Jul 24, 2018 at 7:54 AM Daniel Carrasco >> >> >> >> <d.carra...@i2tic.com> wrote: >> >> >> >> > >> >> >> >> > Yeah, is also my thread. This thread was created before lower the >> >> >> >> > cache size from 512Mb to 8Mb. I thought that maybe was my fault >> >> >> >> > and I did a misconfiguration, so I've ignored the problem until >> >> >> >> > now. >> >> >> >> > >> >> >> >> > Greetings! >> >> >> >> > >> >> >> >> > El mar., 24 jul. 2018 1:00, Gregory Farnum <gfar...@redhat.com> >> >> >> >> > escribió: >> >> >> >> >> >> >> >> >> >> On Mon, Jul 23, 2018 at 11:08 AM Patrick Donnelly >> >> >> >> >> <pdonn...@redhat.com> wrote: >> >> >> >> >>> >> >> >> >> >>> On Mon, Jul 23, 2018 at 5:48 AM, Daniel Carrasco >> >> >> >> >>> <d.carra...@i2tic.com> wrote: >> >> >> >> >>> > Hi, thanks for your response. >> >> >> >> >>> > >> >> >> >> >>> > Clients are about 6, and 4 of them are the most of time on >> >> >> >> >>> > standby. Only two >> >> >> >> >>> > are active servers that are serving the webpage. Also we've a >> >> >> >> >>> > varnish on >> >> >> >> >>> > front, so are not getting all the load (below 30% in PHP is >> >> >> >> >>> > not much). >> >> >> >> >>> > About the MDS cache, now I've the mds_cache_memory_limit at >> >> >> >> >>> > 8Mb. >> >> >> >> >>> >> >> >> >> >>> What! Please post `ceph daemon mds.<name> config diff`, `... >> >> >> >> >>> perf >> >> >> >> >>> dump`, and `... dump_mempools ` from the server the active MDS >> >> >> >> >>> is on. >> >> >> >> >>> >> >> >> >> >>> > I've tested >> >> >> >> >>> > also 512Mb, but the CPU usage is the same and the MDS RAM >> >> >> >> >>> > usage grows up to >> >> >> >> >>> > 15GB (on a 16Gb server it starts to swap and all fails). With >> >> >> >> >>> > 8Mb, at least >> >> >> >> >>> > the memory usage is stable on less than 6Gb (now is using >> >> >> >> >>> > about 1GB of RAM). >> >> >> >> >>> >> >> >> >> >>> We've seen reports of possible memory leaks before and the >> >> >> >> >>> potential >> >> >> >> >>> fixes for those were in 12.2.6. How fast does your MDS reach >> >> >> >> >>> 15GB? >> >> >> >> >>> Your MDS cache size should be configured to 1-8GB (depending on >> >> >> >> >>> your >> >> >> >> >>> preference) so it's disturbing to see you set it so low. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> See also the thread "[ceph-users] Fwd: MDS memory usage is very >> >> >> >> >> high", which had more discussion of that. The MDS daemon >> >> >> >> >> seemingly had 9.5GB of allocated RSS but only believed 489MB was >> >> >> >> >> in use for the cache... >> >> >> >> >> -Greg >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> -- >> >> >> >> >>> Patrick Donnelly >> >> >> >> >>> _______________________________________________ >> >> >> >> >>> ceph-users mailing list >> >> >> >> >>> ceph-users@lists.ceph.com >> >> >> >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> > >> >> >> >> > _______________________________________________ >> >> >> >> > ceph-users mailing list >> >> >> >> > ceph-users@lists.ceph.com >> >> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > _________________________________________ >> >> >> > >> >> >> > Daniel Carrasco Marín >> >> >> > Ingeniería para la Innovación i2TIC, S.L. >> >> >> > Tlf: +34 911 12 32 84 Ext: 223 >> >> >> > www.i2tic.com >> >> >> > _________________________________________ >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > _________________________________________ >> >> > >> >> > Daniel Carrasco Marín >> >> > Ingeniería para la Innovación i2TIC, S.L. >> >> > Tlf: +34 911 12 32 84 Ext: 223 >> >> > www.i2tic.com >> >> > _________________________________________ >> > >> > >> > >> > >> > -- >> > _________________________________________ >> > >> > Daniel Carrasco Marín >> > Ingeniería para la Innovación i2TIC, S.L. >> > Tlf: +34 911 12 32 84 Ext: 223 >> > www.i2tic.com >> > _________________________________________ > > > > > -- > _________________________________________ > > Daniel Carrasco Marín > Ingeniería para la Innovación i2TIC, S.L. > Tlf: +34 911 12 32 84 Ext: 223 > www.i2tic.com > _________________________________________ _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
