Finally found an error that seems to provide some direction:

-1> 2015-03-07 02:52:19.378808 7f175b1cf700  0 log [ERR] : scrub 3.18e
e08a418e/rbd_data.3f7a2ae8944a.00000000000016c8/7//3 on disk size (0) does
not match object info size (4120576) ajusted for ondisk to (4120576)

I'm diving into google now and hoping for something useful. If anyone has a
suggestion, I'm all ears!

QH

On Fri, Mar 6, 2015 at 6:26 PM, Quentin Hartman <
qhart...@direwolfdigital.com> wrote:

> Thanks for the suggestion, but that doesn't seem to have made a difference.
>
> I've shut the entire cluster down and brought it back up, and my config
> management system seems to have upgraded ceph to 0.80.8 during the reboot.
> Everything seems to have come back up, but I am still seeing the crash
> loops, so that seems to indicate that this is definitely something
> persistent, probably tied to the OSD data, rather than some weird transient
> state.
>
>
> On Fri, Mar 6, 2015 at 5:51 PM, Sage Weil <s...@newdream.net> wrote:
>
>> It looks like you may be able to work around the issue for the moment with
>>
>>  ceph osd set nodeep-scrub
>>
>> as it looks like it is scrub that is getting stuck?
>>
>> sage
>>
>>
>> On Fri, 6 Mar 2015, Quentin Hartman wrote:
>>
>> > Ceph health detail - http://pastebin.com/5URX9SsQpg dump summary (with
>> > active+clean pgs removed) - http://pastebin.com/Y5ATvWDZ
>> > an osd crash log (in github gist because it was too big for pastebin) -
>> > https://gist.github.com/qhartman/cb0e290df373d284cfb5
>> >
>> > And now I've got four OSDs that are looping.....
>> >
>> > On Fri, Mar 6, 2015 at 5:33 PM, Quentin Hartman
>> > <qhart...@direwolfdigital.com> wrote:
>> >       So I'm in the middle of trying to triage a problem with my ceph
>> >       cluster running 0.80.5. I have 24 OSDs spread across 8 machines.
>> >       The cluster has been running happily for about a year. This last
>> >       weekend, something caused the box running the MDS to sieze hard,
>> >       and when we came in on monday, several OSDs were down or
>> >       unresponsive. I brought the MDS and the OSDs back on online, and
>> >       managed to get things running again with minimal data loss. Had
>> >       to mark a few objects as lost, but things were apparently
>> >       running fine at the end of the day on Monday.
>> > This afternoon, I noticed that one of the OSDs was apparently stuck in
>> > a crash/restart loop, and the cluster was unhappy. Performance was in
>> > the tank and "ceph status" is reporting all manner of problems, as one
>> > would expect if an OSD is misbehaving. I marked the offending OSD out,
>> > and the cluster started rebalancing as expected. However, I noticed a
>> > short while later, another OSD has started into a crash/restart loop.
>> > So, I repeat the process. And it happens again. At this point I
>> > notice, that there are actually two at a time which are in this state.
>> >
>> > It's as if there's some toxic chunk of data that is getting passed
>> > around, and when it lands on an OSD it kills it. Contrary to that,
>> > however, I tried just stopping an OSD when it's in a bad state, and
>> > once the cluster starts to try rebalancing with that OSD down and not
>> > previously marked out, another OSD will start crash-looping.
>> >
>> > I've investigated the disk of the first OSD I found with this problem,
>> > and it has no apparent corruption on the file system.
>> >
>> > I'll follow up to this shortly with links to pastes of log snippets.
>> > Any input would be appreciated. This is turning into a real cascade
>> > failure, and I haven't any idea how to stop it.
>> >
>> > QH
>> >
>> >
>> >
>> >
>>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to