Re: [ceph-users] OSD mystery

2014-04-01 Thread Reynold PJ
Could clock skew on mon.rts23 cause osd from not listing ? On 04/01/2014 02:28 AM, Dan Koren wrote: > clock skew detected on mon.rts23 signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://l

Re: [ceph-users] OSD mystery

2014-03-31 Thread Dan Koren
Thanks for the prompt reply. The OSDs are set up on dedicated devices, and the mappings are in /etc/fstab. mount shows: /dev/rssda on /var/lib/ceph/osd/ceph-0 type xfs (rw) and similar on all other nodes. Thx, dk On Mon, Mar 31, 2014 at 1:12 PM, Gregory Farnum wrote: > Well, you killed them a

Re: [ceph-users] OSD mystery

2014-03-31 Thread Gregory Farnum
Well, you killed them as part of the reboot...they should have restarted automatically when the system turned on, but that will depend on your configuration and how they were set up. (Eg, if they are each getting a dedicated hard drive, make sure the system knows the drive is present.) What version

Re: [ceph-users] OSD mystery

2014-03-31 Thread Dan Koren
Hi Greg, Thanks for the prompt response. Sure enough, I do see all the OSDs are now down. However, I do not understand the meaning of the sentence about killing the OSDs. This was an OS level reboot of the entire cluster, not issuing any ceph commands either before or after the restart. Doesn't Cep

Re: [ceph-users] OSD mystery

2014-03-31 Thread Gregory Farnum
If you wait longer, you should see the remaining OSDs get marked down. We detect down OSDs in two ways: 1) OSDs heartbeat each other frequently and issue reports when the heartbeat responses take too long. (This is the main way.) 2) OSDs periodically send statistics to the monitors, and if these st

[ceph-users] OSD mystery

2014-03-31 Thread Dan Koren
On a 4 node cluster (admin + 3 mon/osd nodes) I see the following shortly after rebooting the cluster and waiting for a couple of minutes: root@rts23:~# ps -ef | grep ceph && ceph osd tree root 4183 1 0 12:09 ?00:00:00 /usr/bin/ceph-mon --cluster=ceph -i rts23 -f root 577