I would not run the cephfs disaster recovery tools. Your cluster was offline here because it couldn't do some writes, but it should still be self-consistent.
On Thu, Jun 14, 2018 at 4:52 PM Oliver Schulz <oliver.sch...@tu-dortmund.de> wrote: > They are recovered now, looks like it just took a bit > for them to "jump the queue". :-) > > Whew ... > > I rember something about there being some kind of > fschk for CephFS now. Is something like this that I > can/should run before I start my MDS daemons again? > > May then I can finally reduce my MDS max-rank back > to one (I went to two, but it caused trouble, I think > because we have some clients that are too old). > > On 14.06.2018 22:47, Gregory Farnum wrote: > > I don't think there's a way to help them. They "should" get priority in > > recovery, but there were a number of bugs with it in various versions > > and forcing that kind of priority without global decision making is > > prone to issues. > > > > But yep, looks like things will eventually become all good now. :) > > > > On Thu, Jun 14, 2018 at 4:39 PM Oliver Schulz > > <oliver.sch...@tu-dortmund.de <mailto:oliver.sch...@tu-dortmund.de>> > wrote: > > > > Thanks, Greg!! > > > > I reset all the OSD weights to 1.00, and I think I'm in a much > > better state now. The only trouble left in "ceph health detail" is > > > > PG_DEGRADED Degraded data redundancy: 4/404985012 objects degraded > > (0.000%), 3 pgs degraded > > pg 2.47 is active+recovery_wait+degraded+remapped, acting > > [177,68,187] > > pg 2.1fd is active+recovery_wait+degraded+remapped, acting > > [36,83,185] > > pg 2.748 is active+recovery_wait+degraded, acting [31,8,149] > > > > (There's a lot of misplaced PGs now, obviously). The interesting > > thing is that my "lost" PG is back, too, with three acting OSDs. > > > > Maybe I dodged the bullet - what do you think? > > > > One question: Is there a way to give recovery of the three > > degraded PGs priority over backfilling the misplaced ones? > > I tried "ceph pg force-recovery" but it didn't seem to have > > any effect, they were still on "recovery_wait", after. > > > > > > Cheers, > > > > Oliver > > > > > > On 14.06.2018 22:09, Gregory Farnum wrote: > > > On Thu, Jun 14, 2018 at 4:07 PM Oliver Schulz > > > <oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> wrote: > > > > > > Hi Greg, > > > > > > I increased the hard limit and rebooted everything. The > > > PG without acting OSDs still has none, but I also have > > > quite a few PGs with that look like this, now: > > > > > > pg 1.79c is stuck undersized for 470.640254, current > state > > > active+undersized+degraded, last acting [179,154] > > > > > > I had that problem before (only two acting OSDs on a few PGs), > > > I always solved it by setting the primary OSD to out and then > > > back in a few seconds later (resulting in a very quick > recovery, > > > then all was fine again). But maybe that's not the ideal > > solution? > > > > > > Here's "ceph pg map" for one of them: > > > > > > osdmap e526060 pg 1.79c (1.79c) -> up [179,154] acting > > [179,154] > > > > > > I also have two PG's that have only one acting OSD, now: > > > > > > osdmap e526060 pg 0.58a (0.58a) -> up [174] acting [174] > > > osdmap e526060 pg 2.139 (2.139) -> up [61] acting [61] > > > > > > How can I make Ceph assign three OSD's to all of these weird > PGs? > > > Before the reboot, they all did have three OSDs assigned > > (except for > > > the one that has none), and they were not shown as degraded. > > > > > > > > > > If it's the second, then fixing the remapping problem will > > > resolve it. > > > > That's probably/hopefully just by undoing the > > remap-by-utilization > > > > changes. > > > > > > How do I do that, best? Just set all the weights back to 1.00? > > > > > > > > > Yeah. This is probably the best way to fix up the other > > undersized PGs — > > > at least, assuming it doesn't result in an over-full PG! > > > > > > I don't work with overflowing OSDs/clusters often, but my > > suspicion is > > > you're better off with something like CERN's reweight scripts > > than using > > > reweight-by-utilization. Unless it's improved without my > > noticing, that > > > algorithm just isn't very good. :/ > > > -Greg > > > > > > > > > > > > Cheers, > > > > > > Oliver > > > > > > > > > P.S.: Thanks so much for helping! > > > > > > > > > > > > On 14.06.2018 21:37, Gregory Farnum wrote: > > > > On Thu, Jun 14, 2018 at 3:26 PM Oliver Schulz > > > > <oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>> wrote: > > > > > > > > But the contents of the remapped PGs should still be > > > > Ok, right? What confuses me is that they don't > > > > backfill - why don't the "move" where they belong? > > > > > > > > As for the PG hard limit, yes, I ran into this. Our > > > > cluster had been very (very) full, but I wanted the > > > > new OSD nodes to use bluestore, so I updated to > > > > Luminous before I added the additional storage. I > > > > temporarily increased the pg hard limit and after > > > > a while (and after adding the new OSDs) the cluster > > > > seemed to be in a decent state again. Afterwards, > > > > I set the PG hard limit back to normal. > > > > > > > > I don't have a "too many PGs per OSD" health warning, > > > > currently - should I still increase the PG hard limit? > > > > > > > > > > > > Well, it's either the hard limit getting hit, or the fact > that > > > the PG > > > > isn't getting mapped to any OSD and there not being an > > existing > > > primary > > > > to take responsibility for remapping it. > > > > > > > > If it's the second, then fixing the remapping problem will > > > resolve it. > > > > That's probably/hopefully just by undoing the > > > remap-by-utilization changes. > > > > > > > > > > > > On 14.06.2018 20:58, Gregory Farnum wrote: > > > > > Okay, I can’t tell you what happened to that one > > pg, but > > > you’ve got > > > > > another 445 remapped pgs and that’s not a good > > state to be > > > in. It > > > > was > > > > > probably your use of the rewritten-by-utilization. > > :/ I am > > > pretty > > > > sure > > > > > the missing PG and remapped ones have the same root > > cause, > > > and it’s > > > > > possible but by no means certain fixing one will > > fix the > > > others. > > > > > > > > > > > > > > > ...oh, actually, the most likely cause just came up > > in an > > > unrelated > > > > > conversation. You’ve probably run into the pg > overdose > > > protection > > > > that > > > > > was added in luminous. Check the list archives for > > the exact > > > > name, but > > > > > you’ll want to increase the pg hard limit and > > restart the > > > osds that > > > > > exceeded the previous/current setting. > > > > > -Greg > > > > > On Thu, Jun 14, 2018 at 2:33 PM Oliver Schulz > > > > > <oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>>> wrote: > > > > > > > > > > I'm not running the balancer, but I did > > > reweight-by-utilization > > > > > a few times recently. > > > > > > > > > > "ceph osd tree" and "ceph -s" say: > > > > > > > > > > > > > https://gist.github.com/oschulz/36d92af84851ec42e09ce1f3cacbc110 > > > > > > > > > > > > > > > > > > > > On 14.06.2018 20:23, Gregory Farnum wrote: > > > > > > Well, if this pg maps to no osds, something > has > > > certainly > > > > gone wrong > > > > > > with your crush map. What’s the crush rule > it’s > > > using, and > > > > what’s > > > > > the > > > > > > output of “ceph osd tree”? > > > > > > Are you running the manager’s balancer > module or > > > something > > > > that > > > > > might be > > > > > > putting explicit mappings into the osd map > and > > > broken it? > > > > > > > > > > > > I’m not certain off-hand about the pg > > reporting, but I > > > > believe if > > > > > it’s > > > > > > reporting the state as unknown that means > *no* > > > running osd > > > > which > > > > > > contains any copy of that pg. That’s not > > something > > > which ceph > > > > > could do > > > > > > on its own without failures of osds. What’s > the > > > output of > > > > “ceph -s”? > > > > > > On Thu, Jun 14, 2018 at 2:15 PM Oliver Schulz > > > > > > <oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>> > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>>>> wrote: > > > > > > > > > > > > Dear Greg, > > > > > > > > > > > > no, it's a very old cluster (continuous > > operation > > > > since 2013, > > > > > > with multiple extensions). It's a > production > > > cluster and > > > > > > there's about 300TB of valuable data on > it. > > > > > > > > > > > > We recently updated to luminous and > > added more > > > OSDs (a > > > > month > > > > > > ago or so), but everything seemed Ok > > since then. We > > > > didn't have > > > > > > any disk failures, but we had trouble > > with the > > > MDS daemons > > > > > > in the last days, so there were a few > > reboots. > > > > > > > > > > > > Is it somehow possible to find this > > "lost" PG > > > again? Since > > > > > > it's in the metadata pool, large parts > > of our > > > CephFS > > > > directory > > > > > > tree are currently unavailable. I turned > > the MDS > > > > daemons off > > > > > > for now ... > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > Oliver > > > > > > > > > > > > On 14.06.2018 19:59, Gregory Farnum > wrote: > > > > > > > Is this a new cluster? Or did the > > crush map > > > change > > > > somehow > > > > > > recently? One > > > > > > > way this might happen is if CRUSH > > just failed > > > > entirely to > > > > > map a pg, > > > > > > > although I think if the pg exists > > anywhere it > > > > should still be > > > > > > getting > > > > > > > reported as inactive. > > > > > > > On Thu, Jun 14, 2018 at 8:40 AM > > Oliver Schulz > > > > > > > <oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>> > > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>>> > > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>> > > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>> > > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>> > > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de> > > > <mailto:oliver.sch...@tu-dortmund.de > > <mailto:oliver.sch...@tu-dortmund.de>>>>>>> wrote: > > > > > > > > > > > > > > Dear all, > > > > > > > > > > > > > > I have a serious problem with our > > Ceph > > > cluster: > > > > One of our > > > > > > PGs somehow > > > > > > > ended up in this state (reported > > by "ceph > > > > health detail": > > > > > > > > > > > > > > pg 1.XXX is stuck inactive > for > > > ..., current > > > > > state unknown, > > > > > > > last acting [] > > > > > > > > > > > > > > Also, "ceph pg map 1.xxx" reports: > > > > > > > > > > > > > > osdmap e525812 pg 1.721 > > (1.721) -> > > > up [] > > > > acting [] > > > > > > > > > > > > > > I can't use "ceph pg 1.XXX > > query", it just > > > > hangs with > > > > > no output. > > > > > > > > > > > > > > All OSDs are up and in, I have MON > > > quorum, all > > > > other > > > > > PGs seem > > > > > > to be > > > > > > > fine. > > > > > > > > > > > > > > How can diagnose/fix this? > > > Unfortunately, the PG in > > > > > question > > > > > > is part > > > > > > > of the CephFS metadata pool ... > > > > > > > > > > > > > > Any help would be very, very much > > > appreciated! > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Oliver > > > > > > > > > > _______________________________________________ > > > > > > > ceph-users mailing list > > > > > > > ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>> > > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > <mailto:ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>> > > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>>>> > > > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>> > > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > <mailto:ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>> > > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>> > > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com> > > > <mailto:ceph-users@lists.ceph.com > > <mailto:ceph-users@lists.ceph.com>>>>>> > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > > > > > > > > > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
