[ceph-users] Cluster in warn state, not sure what to do next.

2016-07-21 Thread Salwasser, Zac
Hi, I have a cluster that has been in an unhealthy state for a month or so. We realized the OSDs were flapping due to not having user access to enough file handles, but it took us a while to realize this and we appear to have done a lot of damage to the state of the monitor store in the meanti

[ceph-users] Uncompactable Monitor Store at 69GB -- Re: Cluster in warn state, not sure what to do next.

2016-07-21 Thread Salwasser, Zac
Rephrasing for brevity – I have a monitor store that is 69GB and won’t compact any further on restart or with ‘tell compact’. Has anyone dealt with this before? From: "Salwasser, Zac" Date: Thursday, July 21, 2016 at 1:18 PM To: "ceph-users@lists.ceph.com" Cc: "

Re: [ceph-users] Uncompactable Monitor Store at 69GB -- Re: Cluster in warn state, not sure what to do next.

2016-07-21 Thread Salwasser, Zac
. Any ideas how I can investigate that particular osd further? From: Gregory Farnum Date: Thursday, July 21, 2016 at 3:01 PM To: "Salwasser, Zac" Cc: "ceph-users@lists.ceph.com" , "Heller, Chris" Subject: Re: [ceph-users] Uncompactable Monitor Store at 69GB -- R

Re: [ceph-users] Uncompactable Monitor Store at 69GB -- Re: Cluster in warn state, not sure what to do next.

2016-07-21 Thread Salwasser, Zac
-rf the osd data and osd journal directories and then running “ceph-osd -i 53 --mkfs –mkkey”. Should I have done something different? Will this cover “Make sure to do whatever you need for dmcrypt, journals, etc that are specific to your environment.”? All of the other active osds in our tree hav

Re: [ceph-users] Uncompactable Monitor Store at 69GB -- Re: Cluster in warn state, not sure what to do next.

2016-07-25 Thread Salwasser, Zac
Thank you for your help David. For reference, when I originally re-added the osds to the cluster I had set the weight to 0.0 and failed to set the host value. From: David Turner Date: Thursday, July 21, 2016 at 3:24 PM To: "Salwasser, Zac" , "ceph-users@lists.ceph.com"