'ls' on the respective stores in /var/lib/ceph/mon/ceph.X/store.db returns a list of files (i.e. still present), fsck seems fine. I did notice that one of the nodes has different contents in the /var/lib/ceph/mon/ceph-b/keyring i.e. its key is different from the other 2 nodes'. That shouldn't be the case, should it? Would scp'ing over one of the other node's keyring files while mon.b is stopped be the right course of action then?
Also your red herring explanation... how do I put this... It seems like an important thing to know, so thanks for that. I'm not sure how one would go about putting this tidbit in a spot where people would find it when needed... maybe somewhere in the debugging section of the wiki? On 29/04/2014 18:25, Gregory Farnum wrote: > Monitor keys don't change; I think something else must be going on. Did you > remove any of their stores? Are the local filesystems actually correct > (fsck)? > > The ceph-create-keys is a red herring and will stop as soon as. The > monitors do get into a quorum. > -Greg > > On Tuesday, April 29, 2014, Marc <m...@shoowin.de> wrote: > >> Hi, >> >> still working on a troubled ceph cluster running .61.2-1raring >> consisting of (currently) 4 monitors a,b,c,g with g being a newly added >> monitor that failed/fails to sync up, so consider that one down. Now mon >> a and b died because for some (currently unknown) reason linux created a >> core dump on the root partition (/core) that filled up the partition to >> 0b left and consequently the mons died. Now I tried restarting them, but >> they they seem deadlocked in the following situation: >> >> the corresponding ceph-mon.X logs show various errors about cephx like >> >> "cephx: verify_authorizer could not decrypt ticket info: error: NSS AES >> final round failed: -8190" >> >> "cephx: verify_reply coudln't decrypt with error: error decoding block >> for decryption" >> >> I can see that the /usr/sbin/ceph-create-keys process is stuck (based on >> the fact that its still running 20 minutes later). Manually running this >> says: >> >> >> INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing' >> >> >> >> So, the monitors dont start up (stuck probing) because they cant >> communicate because they need new keys, and the keys cannot be generated >> because theres no quorum. Is there a way to fix this? >> >> >> Kind regards, >> Marc >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com <javascript:;> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
