On Wed, Sep 4, 2013 at 11:21 AM, Bernhard Glomm <bernhard.gl...@ecologic.eu>wrote:
> Removing a monitor, also like described in > http://ceph.com/docs/master/rados/deployment/ceph-deploy-mon/ > doesn't work > > ---------------- > root@nuke36[/1]:/etc/ceph # ceph-deploy mon destroy nuke36 > [ceph_deploy.mon][DEBUG ] Removing mon from nuke36 > [ceph_deploy.mon][ERROR ] ceph-mon deamon did not stop > [ceph_deploy][ERROR ] GenericError: Failed to destroy 1 monitors > ---------------- > > is this, I can't destroy the mon on which I run the destroy command? > (I could destroy > but since I just added another two monitors to my cluster, > the broken first one is alive again... > What OS is the nuke36 host? There are a bunch of commands that `mon destroy` will call on the remote host, but unfortunately, this is one of the commands I have not fixed yet to use the new remote logging to tell you. Can you try to manually run this on nuke36 and see what is going on? sudo ceph --cluster={cluster name, defaults to ceph} -n mon. -k {path to keyring}/keyring mon remove > > well that should do as a work around for now > (since I'm using the dev branch and do only testing stuff) > still, would be curious if I could have repaired it any other way. > > Bernhard > > Am 04.09.2013 12:40:04, schrieb Bernhard Glomm: > > Hi all, > > after some days of successful creating and destroying rbd's, snapshots, > clones and migrating formats all of a sudden one of the monitors doesn't > work anymore. > I tried to remove and re-add the monitor from the cluster, but that > doesn't seem to work either. > > Here's the situation: > I liked to use ceph-deploy to initiate the cluster. > Due to the broken ceph-create-keys in dumpling I turned to the gitbuilder > version > now running > > ceph version 0.67.2-23-g24f2669 (24f2669783e2eb9d9af5ecbe106efed93366ba63) > on uptodate raring systems > All of a sudden the host from which I ran ceph-deploy, and which should be > one of the > 5 monitors (from which 2 are also serving as OSDs) has fallen out of the > quorum as you can see here > (yes, time is in sync on all nodes): > > ------------------ > root@nuke36[/0]:~ # ceph -s > 2013-09-04 11:09:56.039547 7f7a8820f700 1 -- :/0 messenger.start > 2013-09-04 11:09:56.040646 7f7a8820f700 1 -- :/1016260 --> > 192.168.242.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 > 0x7f7a8000e8f0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.041304 7f7a84a08700 1 -- 192.168.242.36:0/1016260learned > my addr > 192.168.242.36:0/1016260 > 2013-09-04 11:09:56.042843 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 1 ==== mon_map v1 ==== 776+0+0 (2241333437 0 0) > 0x7f7a70000c30 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.043038 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 > (2063715990 0 0) 0x7f7a70001060 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.043324 7f7a86a0c700 1 -- 192.168.242.36:0/1016260--> > 192.168.242.92:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 > 0x7f7a74001af0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.044197 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== > 206+0+0 (3910749728 0 0) 0x7f7a70001060 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.044375 7f7a86a0c700 1 -- 192.168.242.36:0/1016260--> > 192.168.242.92:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 > 0x7f7a740020d0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.045376 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== > 393+0+0 (3802320753 0 0) 0x7f7a700008f0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.045457 7f7a86a0c700 1 -- 192.168.242.36:0/1016260--> > 192.168.242.92:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 > 0x7f7a8000ed80 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.045550 7f7a8820f700 1 -- 192.168.242.36:0/1016260--> > 192.168.242.92:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 > 0x7f7a800079f0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.045559 7f7a8820f700 1 -- 192.168.242.36:0/1016260--> > 192.168.242.92:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 > 0x7f7a8000fa10 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.046376 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 5 ==== mon_map v1 ==== 776+0+0 (2241333437 0 0) > 0x7f7a70001290 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.046417 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 > (1524320885 0 0) 0x7f7a70001480 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.046429 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 7 ==== osd_map(22..22 src has 1..22) v3 ==== > 2355+0+0 (11792226 0 0) 0x7f7a70001f70 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.046828 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 > (1524320885 0 0) 0x7f7a700008f0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.046948 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 9 ==== osd_map(22..22 src has 1..22) v3 ==== > 2355+0+0 (11792226 0 0) 0x7f7a700008c0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.047071 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 10 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 > (1524320885 0 0) 0x7f7a70000df0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.047547 7f7a8820f700 1 -- 192.168.242.36:0/1016260--> > 192.168.242.92:6789/0 -- mon_command({"prefix": > "get_command_descriptions"} v 0) v1 -- ?+0 0x7f7a8000b0f0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.050938 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 11 ==== mon_command_ack([{"prefix": > "get_command_descriptions"}]=0 v0) v1 ==== 72+0+24040 (1092875540 0 > 2922658865) 0x7f7a700008c0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.089981 7f7a8820f700 1 -- 192.168.242.36:0/1016260--> > 192.168.242.92:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- ?+0 > 0x7f7a8000b0d0 con 0x7f7a8000e4e0 > 2013-09-04 11:09:56.091348 7f7a86a0c700 1 -- 192.168.242.36:0/1016260<== > mon.3 > 192.168.242.92:6789/0 12 ==== mon_command_ack([{"prefix": "status"}]=0 > v0) v1 ==== 54+0+558 (1155462804 0 1174924833) 0x7f7a70000db0 con > 0x7f7a8000e4e0 > cluster b085fba3-8e17-443c-bb61-7758504538f8 > health HEALTH_WARN 1 mons down, quorum 0,1,3,4 atom01,atom02,ping,pong > monmap e1: 5 mons at {atom01= > 192.168.242.31:6789/0,atom02=192.168.242.32:6789/0,nuke36=192.168.242.36:6789/0,ping=192.168.242.92:6789/0,pong=192.168.242.93:6789/0}, > election epoch 26, quorum 0,1,3,4 atom01,atom02,ping,pong > osdmap e22: 2 osds: 2 up, 2 in > pgmap v46761: 1192 pgs: 1192 active+clean; 7806 MB data, 20618 MB > used, 3702 GB / 3722 GB avail; 9756B/s wr, 0op/s > mdsmap e17: 1/1/1 up {0=pong=up:active}, 1 up:standby > > 2013-09-04 11:09:56.096071 7f7a8820f700 1 -- > 192.168.242.36:0/1016260mark_down 0x7f7a8000e4e0 -- 0x7f7a8000e280 > 2013-09-04 11:09:56.096516 7f7a8820f700 1 -- > 192.168.242.36:0/1016260mark_down_all > 2013-09-04 11:09:56.097065 7f7a8820f700 1 -- > 192.168.242.36:0/1016260shutdown complete. > ------------------ > > this is the ceph.conf that was generated during ceph-deploy (well I added > the debug lines obviously) > > ------------------ > root@nuke36[/0]:~ # cat /etc/ceph/ceph.conf > [global] > fsid = b085fba3-8e17-443c-bb61-7758504538f8 > mon_initial_members = ping, pong, nuke36, atom01, atom02 > mon_host = > 192.168.242.92,192.168.242.93,192.168.242.36,192.168.242.31,192.168.242.32 > auth_supported = cephx > osd_journal_size = 1024 > filestore_xattr_use_omap = true > debug ms = 1 > debug mon = 20 > ------------------ > > after a reboot of the node I now find > > ------------------ > root@nuke36[/1]:~ # ps ax | egrep ceph > 855 ? Ssl 0:00 /usr/bin/ceph-mon --cluster=ceph -i nuke36 -f > 856 ? Ss 0:00 /usr/bin/python /usr/sbin/ceph-create-keys > --cluster=ceph -i nuke36 > 1813 pts/1 R+ 0:00 egrep --color=auto ceph > ------------------ > > with ceph-create-keys runing infinetly, > while the key files but already exist > > ------------------ > root@nuke36[/1]:~ # ls -lh /etc/ceph/ > total 88K > -rw-r--r-- 1 root root 72 Aug 30 15:54 ceph.bootstrap-mds.keyring > -rw-r--r-- 1 root root 72 Aug 30 15:54 ceph.bootstrap-osd.keyring > -rw------- 1 root root 64 Aug 30 15:54 ceph.client.admin.keyring > -rw-r--r-- 1 root root 303 Sep 4 10:19 ceph.conf > -rw-r--r-- 1 root root 59K Sep 3 10:15 ceph.log > -rw-r--r-- 1 root root 73 Aug 30 15:53 ceph.mon.keyring > -rw-r--r-- 1 root root 92 Aug 30 00:03 rbdmap > ------------------ > > and > > ------------------ > root@nuke36[/1]:~ # tree -pfugiAD /var/lib/ceph/ > /var/lib/ceph > [drwxr-xr-x root root Aug 30 15:54] /var/lib/ceph/bootstrap-mds > [-rw------- root root Aug 30 15:54] > /var/lib/ceph/bootstrap-mds/ceph.keyring > [drwxr-xr-x root root Aug 30 15:54] /var/lib/ceph/bootstrap-osd > [-rw------- root root Aug 30 15:54] > /var/lib/ceph/bootstrap-osd/ceph.keyring > [drwxr-xr-x root root Aug 30 0:00] /var/lib/ceph/mds > [drwxr-xr-x root root Aug 30 15:53] /var/lib/ceph/mon > [drwxr-xr-x root root Aug 30 15:53] /var/lib/ceph/mon/ceph-nuke36 > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-nuke36/done > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-nuke36/keyring > [drwxr-xr-x root root Sep 4 11:16] > /var/lib/ceph/mon/ceph-nuke36/store.db > [-rw-r--r-- root root Sep 3 6:21] > /var/lib/ceph/mon/ceph-nuke36/store.db/007726.sst > [-rw-r--r-- root root Sep 3 6:21] > /var/lib/ceph/mon/ceph-nuke36/store.db/007727.sst > [-rw-r--r-- root root Sep 3 6:21] > /var/lib/ceph/mon/ceph-nuke36/store.db/007728.sst > [-rw-r--r-- root root Sep 3 6:21] > /var/lib/ceph/mon/ceph-nuke36/store.db/007729.sst > [-rw-r--r-- root root Sep 3 6:21] > /var/lib/ceph/mon/ceph-nuke36/store.db/007730.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007767.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007768.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007769.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007770.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007772.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007773.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007774.sst > [-rw-r--r-- root root Sep 4 10:14] > /var/lib/ceph/mon/ceph-nuke36/store.db/007775.sst > [-rw-r--r-- root root Sep 4 10:18] > /var/lib/ceph/mon/ceph-nuke36/store.db/007777.sst > [-rw-r--r-- root root Sep 4 10:19] > /var/lib/ceph/mon/ceph-nuke36/store.db/007780.sst > [-rw-r--r-- root root Sep 4 11:16] > /var/lib/ceph/mon/ceph-nuke36/store.db/007783.sst > [-rw-r--r-- root root Sep 4 11:16] > /var/lib/ceph/mon/ceph-nuke36/store.db/007784.log > [-rw-r--r-- root root Sep 4 11:16] > /var/lib/ceph/mon/ceph-nuke36/store.db/CURRENT > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-nuke36/store.db/LOCK > [-rw-r--r-- root root Sep 4 11:16] > /var/lib/ceph/mon/ceph-nuke36/store.db/LOG > [-rw-r--r-- root root Sep 4 10:19] > /var/lib/ceph/mon/ceph-nuke36/store.db/LOG.old > [-rw-r--r-- root root Sep 4 11:16] > /var/lib/ceph/mon/ceph-nuke36/store.db/MANIFEST-007782 > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-nuke36/upstart > [drwxr-xr-x root root Aug 30 0:00] /var/lib/ceph/osd > [drwxr-xr-x root root Aug 30 15:53] /var/lib/ceph/tmp > ------------------ > > to compare with atom01 which is still running in the cluster... > > ------------------ > root@atom01[/0]:~ # tree -pfugiAD /var/lib/ceph/ > /var/lib/ceph > [drwxr-xr-x root root Aug 30 15:54] /var/lib/ceph/bootstrap-mds > [-rw------- root root Aug 30 15:54] > /var/lib/ceph/bootstrap-mds/ceph.keyring > [drwxr-xr-x root root Aug 30 15:54] /var/lib/ceph/bootstrap-osd > [-rw------- root root Aug 30 15:54] > /var/lib/ceph/bootstrap-osd/ceph.keyring > [drwxr-xr-x root root Aug 30 0:00] /var/lib/ceph/mds > [drwxr-xr-x root root Aug 30 15:53] /var/lib/ceph/mon > [drwxr-xr-x root root Aug 30 15:53] /var/lib/ceph/mon/ceph-atom01 > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-atom01/done > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-atom01/keyring > [drwxr-xr-x root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db > [-rw-r--r-- root root Sep 4 11:25] > /var/lib/ceph/mon/ceph-atom01/store.db/006339.log > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/006342.sst > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/006343.sst > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/006344.sst > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/006345.sst > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/006346.sst > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/006347.sst > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/006348.sst > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-atom01/store.db/CURRENT > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-atom01/store.db/LOCK > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/LOG > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-atom01/store.db/LOG.old > [-rw-r--r-- root root Sep 4 11:18] > /var/lib/ceph/mon/ceph-atom01/store.db/MANIFEST-000004 > [-rw-r--r-- root root Aug 30 15:53] > /var/lib/ceph/mon/ceph-atom01/upstart > [drwxr-xr-x root root Aug 30 0:00] /var/lib/ceph/osd > [drwxr-xr-x root root Aug 30 15:53] /var/lib/ceph/tmp > ------------------ > > the problem with ceph-create-keys seemed to have been fixed with wip-4924 > since in that version "rbd create" wouldn't work I switched to > deb http://gitbuilder.ceph.com/ceph-deb-raring-x86_64-basic/ref/dumpling/ > raring main > (runing on an uptodate raring) > I thought of just removing and than re-adding the failing mon but who do I > do that? > The documentation on > http://ceph.com/docs/master/rados/operations/add-or-rm-mons/ > says: > "service ceph -a stop mon.{mon-id}" > > ------------------ > root@nuke36[/1]:~ # service ceph -a stop mon.nuke36 > /etc/init.d/ceph: mon.nuke36 not found (/etc/ceph/ceph.conf defines , > /var/lib/ceph defines ) > root@nuke36[/1]:~ # service ceph -a stop mon.c > /etc/init.d/ceph: mon.c not found (/etc/ceph/ceph.conf defines , > /var/lib/ceph defines ) > root@nuke36[/1]:~ # service ceph -a stop mon.2 > /etc/init.d/ceph: mon.2 not found (/etc/ceph/ceph.conf defines , > /var/lib/ceph defines ) > root@nuke36[/1]:~ # service ceph -a stop mon.ceph-nuke36 > /etc/init.d/ceph: mon.ceph-nuke36 not found (/etc/ceph/ceph.conf defines , > /var/lib/ceph defines ) > ------------------ > > that doesn't help, manually stopping the daemon doesn't work either > (respawning, okay) > but this combination leaves me quite curious > > ------------------ > root@atom01[/0]:~ # ceph health detail > HEALTH_WARN 1 mons down, quorum 0,1,3,4 atom01,atom02,ping,pong > mon.nuke36 (rank 2) addr 192.168.242.36:6789/0 is down (out of quorum) > root@atom01[/0]:~ # service ceph -a stop mon.nuke36 > /etc/init.d/ceph: mon.nuke36 not found (/etc/ceph/ceph.conf defines , > /var/lib/ceph defines ) > /etc/init.d/ceph: mon.2 not found (/etc/ceph/ceph.conf defines , > /var/lib/ceph defines ) > root@atom01[/0]:~ # service ceph -a stop mon.rank2 > ------------------ > > something seems to be out of sync, at least with the documentation? > any hint how to proceed from here? > > TIA > > Bernhard > > > -- > ------------------------------ > > *Bernhard Glomm* > IT Administration > > Phone: +49 (30) 86880 134 Fax: +49 (30) 86880 100 Skype: > bernhard.glomm.ecologic > <http://ecologic.eu> <http://www.youtube.com/v/hZtiK04A9Yo> > <http://ecologic.eu/newsletter/subscribe> > <http://www.facebook.com/Ecologic.Institute> > <http://www.linkedin.com/company/ecologic-institute-berlin-germany> > <http://twitter.com/EcologicBerlin> > <http://www.youtube.com/user/EcologicInstitute> > <http://plus.google.com/113756356645020994482> Ecologic Institut > gemeinnützige GmbH | Pfalzburger Str. 43/44 | 10717 Berlin | Germany > GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.: > DE811963464 > Ecologic™ is a Trade Mark (TM) of Ecologic Institut gemeinnützige GmbH > ------------------------------ > root@nuke > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com <#140e99031b451ae9_> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > -- > ------------------------------ > [image: *Ecologic Institute*] *Bernhard Glomm* > IT Administration > > Phone: +49 (30) 86880 134 Fax: +49 (30) 86880 100 Skype: > bernhard.glomm.ecologic [image: > Website:] <http://ecologic.eu> [image: | > Video:]<http://www.youtube.com/v/hZtiK04A9Yo> [image: > | Newsletter:] <http://ecologic.eu/newsletter/subscribe> [image: | > Facebook:] <http://www.facebook.com/Ecologic.Institute> [image: | > Linkedin:]<http://www.linkedin.com/company/ecologic-institute-berlin-germany> > [image: > | Twitter:] <http://twitter.com/EcologicBerlin> [image: | > YouTube:]<http://www.youtube.com/user/EcologicInstitute> [image: > | Google+:] <http://plus.google.com/113756356645020994482> Ecologic > Institut gemeinnützige GmbH | Pfalzburger Str. 43/44 | 10717 Berlin | > Germany > GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.: > DE811963464 > Ecologic™ is a Trade Mark (TM) of Ecologic Institut gemeinnützige GmbH > ------------------------------ > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
