The Mon store is important and since your cluster isn't healthy, they need to
hold onto it to make sure that when things come up that the mon can replay
everything for them. Once you fix the 2 down and peering PGs, The mon store
will fix itself in no time at all. Ceph is rightly refusing to compact that
database until your cluster is healthy.
It seems like you have a couple things that might help your setup. First I see
something very easy to resolve, and that's the blocked requests. Try running
the following command:
ceph osd down 71
That command will tell the cluster that osd.71 is down without restarting the
actual osd daemon. Osd.71 will come back and tell the mons it's actually up,
but in the mean time the operations blocking on osd.71 will go to a secondary
to get the response and clear up.
Second, osd.53 looks to be causing the never ending peering. A couple
questions to check things here. What is your osd_max_backfills set to? That
is directly related to how fast osd.53 will fill back up. Something you might
do to speed that up is to just inject a higher setting for osd.53 and not the
rest of the cluster:
ceph tell osd.53 injectargs '--osd_max_backfills=20'
If this is the problem and the cluster is just waiting for osd.53 to finish
backfilling, then this will get you there faster. I'm unfamiliar with the
strategy you used to rebuild the data for osd.53. I would have removed the osd
from the cluster and added it back in with the same weight. That way the osd
would start right away and you would see the pgs backfilling onto the osd as
opposed to it sitting in a perpetual "booting" state.
To remove the osd with minimal impact to the cluster, the following commands
should get you there.
ceph osd tree | grep 'osd.53 '
ceph osd set nobackfill
ceph osd set norecover
#on the host with osd.53, stop the daemon
ceph osd down 53
ceph osd out 53
ceph osd crush remove osd.53
ceph auth rm osd.53
ceph osd rm 53
At this point osd.53 is completely removed from the cluster and you have the
original weight of the osd to set it to when you bring the osd back in. The
down and peering PGs should now be resolved. Now, completely re-format and add
the osd back into the cluster. Make sure to do whatever you need for dmcrypt,
journals, etc that are specific to your environment. Once the osd is back in
the cluster, up and in, reweight the osd to what it was before you removed it
and unset norecover and nobackfill.
ceph osd crush reweight osd.53 {{ weight_from_tree_command }}
ceph osd unset nobackfill
ceph osd unset norecover
At this point everything is back to the way it was and the osd should start
receiving data. The only data movement should be refilling osd.53 with the
data it used to have and everything else should stay the same. Increasing the
backfills for this osd will help it fill up faster, but it will be slower for
client io if you do. The mon stores will remain "too big" until after
backfilling onto osd.53 finishes, but once the data stops moving around and all
of your osds are up and in, the mon stores will compact in no time.
I hope this helps. Ask questions if you have any, and never run a command on
your cluster that you don't understand.
David Turner
________________________________
From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Salwasser,
Zac [zsalw...@akamai.com]
Sent: Thursday, July 21, 2016 12:54 PM
To: ceph-users@lists.ceph.com
Cc: Heller, Chris
Subject: [ceph-users] Uncompactable Monitor Store at 69GB -- Re: Cluster in
warn state, not sure what to do next.
Rephrasing for brevity – I have a monitor store that is 69GB and won’t compact
any further on restart or with ‘tell compact’. Has anyone dealt with this
before?
From: "Salwasser, Zac" <zsalw...@akamai.com>
Date: Thursday, July 21, 2016 at 1:18 PM
To: "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com>
Cc: "Salwasser, Zac" <zsalw...@akamai.com>, "Heller, Chris" <chel...@akamai.com>
Subject: Cluster in warn state, not sure what to do next.
Hi,
I have a cluster that has been in an unhealthy state for a month or so. We
realized the OSDs were flapping due to not having user access to enough file
handles, but it took us a while to realize this and we appear to have done a
lot of damage to the state of the monitor store in the meantime.
I’ve been trying to tackle one issue at a time, starting with the size of the
monitor store. Compaction, either compact on restart or compact as a ‘tell’
operation, does not shrink the size of the monitor store any more than it
presently is. Having no luck getting the monitor store to shrink, I switched
gears to troubleshooting down placement groups. There are two remaining that I
cannot fix, and they both claim to be blocked from peering by the same osd
(osd.53).
Two days ago, I removed the osd data for osd.53 and restarted it after a ‘mkfs’
operation. It has been in the “booting” state ever since, although there is
now 72GB of data in the osd data partition for osd.53, indicating that some
sort of partial “backfilling” has taken place. Watching the host file system
indicates that any data coming into that partition at this point is only
trickling in.
Here is the output of “ceph health detail”. I’m wondering if anyone would be
willing to engage with me to at least get me unstuck. I am on #ceph as
salwasser.
* * *
HEALTH_WARN 2 pgs down; 2 pgs peering; 2 pgs stuck inactive; 2 pgs stuck
unclean; 15 requests are blocked > 32 sec; 1 osds have slow requests; mds0:
Behind on trimming (367/30); mds-1: Behind on trimming (364/30);
mon.a65-121-158-160 store is getting too big! 74468 MB >= 15360 MB;
mon.a65-121-158-161 store is getting too big! 73881 MB >= 15360 MB;
mon.a65-121-158-195 store is getting too big! 64963 MB >= 15360 MB;
mon.a65-121-158-196 store is getting too big! 64023 MB >= 15360 MB;
mon.a65-121-158-197 store is getting too big! 63632 MB >= 15360 MB
pg 4.285 is stuck inactive since forever, current state down+peering, last
acting [28,122,114]
pg 1.716 is stuck inactive for 969017.268003, current state down+peering, last
acting [71,213,55]
pg 4.285 is stuck unclean since forever, current state down+peering, last
acting [28,122,114]
pg 1.716 is stuck unclean for 969351.417382, current state down+peering, last
acting [71,213,55]
pg 1.716 is down+peering, acting [71,213,55]
pg 4.285 is down+peering, acting [28,122,114]
5 ops are blocked > 4194.3 sec
10 ops are blocked > 2097.15 sec
5 ops are blocked > 4194.3 sec on osd.71
10 ops are blocked > 2097.15 sec on osd.71
1 osds have slow requests
mds0: Behind on trimming (367/30)(max_segments: , num_segments: o)
mds-1: Behind on trimming (364/30)(max_segments: , num_segments: l)
mon.a65-121-158-160 store is getting too big! 74468 MB >= 15360 MB -- 53% avail
mon.a65-121-158-161 store is getting too big! 73881 MB >= 15360 MB -- 73% avail
mon.a65-121-158-195 store is getting too big! 64963 MB >= 15360 MB -- 81% avail
mon.a65-121-158-196 store is getting too big! 64023 MB >= 15360 MB -- 81% avail
mon.a65-121-158-197 store is getting too big! 63632 MB >= 15360 MB -- 81% avail
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
