Hi,
we recently migrated some of our nodes to Ubuntu 12, which helped everything
quite a bit.
But we hit a snag where the upstart initscript would not set the file open
ulimit correctly and some OSDs ran out of fds.
Some roblems manifested since then on the node where this happened such as
scrub errors (which were corrected - don't ask me how, I was sleeping :-)) -
but not two of the OSDs on this node started failing with SIGABORT:
2015-09-25 10:45:17.860461 361ea17e700 -1 osd.12 pg_epoch: 1209679 pg[4.3d85( v
1209679'13913518 (1209090'13910508,1209679'13913518] local-les=1209679 n=235
ec=3 les/c 1209679/1209679 1209678/1209678/1209678) [12,36,59] r=0 lpr=1209678
mlcod 1209656'13913517 active+clean snaptrimq=[857c0~1,857f4~1,85a43~2]]
trim_objectcould not find coid
783dbd85/rbd_data.1a785181f15746a.00000000000238df/857c0//4
2015-09-25 10:45:17.862019 361ea17e700 -1 osd/ReplicatedPG.cc: In function
'ReplicatedPG::RepGather* ReplicatedPG::trim_object(const hobject_t&)' thread
361ea17e700 time 2015-09-25 10:45:17.860501
osd/ReplicatedPG.cc: 1510: FAILED assert(0)
ceph version 0.67.11-82-ge5b6eea (e5b6eea91cc37434f78a987d2dd1d3edd4a23f3f)
1: (ReplicatedPG::trim_object(hobject_t const&)+0x150) [0x6e8bd0]
2: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim const&)+0x2e7)
[0x6ee0d7]
3: (boost::statechart::detail::reaction_result
boost::statechart::simple_state<ReplicatedPG::TrimmingObjects,
ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na>,
(boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<ReplicatedPG::SnapTrim>,
boost::statechart::transition<ReplicatedPG::Reset, ReplicatedPG::NotTrimming,
boost::statechart::detail::no_context<ReplicatedPG::Reset>,
&(boost::statechart::detail::no_context<ReplicatedPG::Reset>::no_function(ReplicatedPG::Reset
const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na>,
boost::statechart::simple_state<ReplicatedPG::TrimmingObjects,
ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na>, (boost::statechart::history_mode)0>
>(boost::statechart::simple_state<ReplicatedPG::TrimmingObjects,
ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na>, (boost::statechart::history_mode)0>&, boost::statechart::event_base
const&, void const*)+0x96) [0x740fa6]
4: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer,
ReplicatedPG::NotTrimming, std::allocator<void>,
boost::statechart::null_exception_translator>::process_queued_events()+0x137)
[0x71bdf7]
5: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer,
ReplicatedPG::NotTrimming, std::allocator<void>,
boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base
const&)+0x26) [0x71cfe6]
6: (ReplicatedPG::snap_trimmer()+0x4ed) [0x6b59ad]
7: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x790c54]
8: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68c) [0x9a69cc]
9: (ThreadPool::WorkThread::entry()+0x10) [0x9a7c20]
10: (()+0x7e9a) [0x36258121e9a]
11: (clone()+0x6d) [0x3625669638d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
(full log available on request, file system will be thrashed shortly so tell me
if it's helpful to look for something in there)
Could this all be caused by the OSD running out of file descriptors? Is it
supposed to handle a problem like this (meaning both the assert that happened
now and the file descriptor limit) gracefuly? Or is it a known issue that this
could happen?
The thing about upstart is it apparently keeps restarting the OSD, which makes
the problem even worse.
Luckily we caught this in time and it only happened on one node, so we are
thrashing all the OSDs here.
Looks like a problem that could hit anyone, and if it actually damages data
then it could be pretty bad and maybe worth looking into - tell me what more is
needed.
Config:
XFS filesystem
Ubuntu 12 with 3.14.37 kernel
FIEMAP disabled
Ceph Dumpling (0.67.11-82-ge5b6eea)
Thanks
Jan
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
