Good to see 2 OSD UP and 2 OSD IN :-)
Now with respect to your questions i just know one thing.
admin key are used by admin node and your Ceph Nodes so that you can use the
ceph CLI without having to specify the monitor address and
ceph.client.admin.keyring each time you execute a command.
If you got to know some info on keyrings then let me know.
Regards
karan
----- Original Message -----
From: "Nabil Naim" <nabil_n...@linkdatacenter.net>
To: "Karan Singh" <ksi...@csc.fi>
Cc: ceph-users@lists.ceph.com
Sent: Wednesday, 30 October, 2013 3:16:16 PM
Subject: RE: [ceph-users] activate disk error
:-))))
Yes its firewall issue , I recheck the firewall and disable it currently and it
works :-( sorry for my stupid
The activation shows
[ceph@ceph-deploy my-cluster]$ ceph-deploy osd activate ceph-server02:/dev/sdb1
[ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate
ceph-server02:/dev/sdb1
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph-server02:/dev/sdb1:
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final
[ceph_deploy.osd][DEBUG ] activating host ceph-server02 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[ceph-server02][INFO ] Running command: ceph-disk-activate --mark-init
sysvinit --mount /dev/sdb1
[ceph-server02][INFO ] === osd.1 ===
[ceph-server02][INFO ] Starting Ceph osd.1 on ceph-server02...
[ceph-server02][INFO ] starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1
/var/lib/ceph/osd/ceph-1/journal
[ceph-server02][ERROR ] got latest monmap
[ceph-server02][ERROR ] HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl
for device
[ceph-server02][ERROR ] 2013-10-30 12:22:21.809843 7fc7fe1eb7a0 -1 journal
check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected
3b570378-cd08-4a09-87f7-de6666d1aa1c, invalid (someone else's?) journal
[ceph-server02][ERROR ] HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl
for device
[ceph-server02][ERROR ] HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl
for device
[ceph-server02][ERROR ] HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl
for device
[ceph-server02][ERROR ] 2013-10-30 12:22:21.895490 7fc7fe1eb7a0 -1
filestore(/var/lib/ceph/tmp/mnt.dZI3ak) could not find
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
[ceph-server02][ERROR ] 2013-10-30 12:22:22.036464 7fc7fe1eb7a0 -1 created
object store /var/lib/ceph/tmp/mnt.dZI3ak journal
/var/lib/ceph/tmp/mnt.dZI3ak/journal for osd.1 fsid
40d40711-3884-441e-bf9a-2ea467cebeac
[ceph-server02][ERROR ] 2013-10-30 12:22:22.036539 7fc7fe1eb7a0 -1 auth: error
reading file: /var/lib/ceph/tmp/mnt.dZI3ak/keyring: can't open
/var/lib/ceph/tmp/mnt.dZI3ak/keyring: (2) No such file or directory
[ceph-server02][ERROR ] 2013-10-30 12:22:22.036782 7fc7fe1eb7a0 -1 created new
key in keyring /var/lib/ceph/tmp/mnt.dZI3ak/keyring
[ceph-server02][ERROR ] added key for osd.1
[ceph-server02][ERROR ] create-or-move updating item name 'osd.1' weight 0.02
at location {host=ceph-server02,root=default} to crush map
Is that something wrong ?
now From ceph-node2
[ceph@ceph-node2 ceph]$ sudo ceph status
cluster 40d40711-3884-441e-bf9a-2ea467cebeac
health HEALTH_OK
monmap e1: 1 mons at {ceph-server01=192.168.115.91:6789/0}, election epoch
1, quorum 0 ceph-node1
osdmap e18: 2 osds: 2 up, 2 in
pgmap v31: 192 pgs: 192 active+clean; 0 bytes data, 71308 KB used, 38820 MB
/ 38889 MB avail
mdsmap e1: 0/0/1 up
last question,
ceph-node1(Monitor):/etc/ceph/ceph.client.admin.keyring is identical with
ceph-node2(OSD):/etc/ceph/ceph.client.admin.keyring at ceph-node2(OSD)
also
ceph-node1(Monitor): /var/lib/ceph/bootstrap-osd/ceph.keyring is identical with
cepg-node2(OSD): /var/lib/ceph/bootstrap-osd/ceph.keyring
and the key ring of /var/lib/ceph/osd/ceph-1/keyring doesn't match with any of
them, what is the usage of
/etc/ceph/ceph.client.admin.keyring
And
/var/lib/ceph/bootstrap-osd/ceph.keyring
And
/var/lib/ceph/osd/ceph-1/keyring
And when it should match and when not ?
If you can refer me to article explain functionality of each keyring if that
will take time :-)
Thank you again
Regards,
Nabil Naim
-----Original Message-----
From: Karan Singh [mailto:ksi...@csc.fi]
Sent: Wednesday, October 30, 2013 2:37 PM
To: Nabil Naim
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] activate disk error
Hi Nabil
1) I hope you have taken ceph services bounce back after copying keyring files
2) from your OSD node ( ceph-node2 ) are you able to check your cluster status
#ceph status , it should return output similarly to ceph-node1 ( monitor node
) if not then there is connectivity problem between two.
3) check for iptables between machines ( if this is your testing cluster
disable iptables )
Note: in my setup and installation guide, the Admin node(ceph-deploy) is a
separate server other than ceph-node1(Monitor) and ceph-node2, ceph-node3(OSD),
and the admin node doesn't required ceph installation only ceph-deploy, is that
right ? also admin node (ceph-deploy) gather keys from ceph-node1 (monitor
node) only, right ?
yes this seems to be right
4) also check copying keys from ceph-deploy node to ceph-node2
Regards
Karan
----- Original Message -----
From: "Nabil Naim" <nabil_n...@linkdatacenter.net>
To: "Karan Singh" <ksi...@csc.fi>
Cc: ceph-users@lists.ceph.com
Sent: Wednesday, 30 October, 2013 1:24:33 PM
Subject: RE: [ceph-users] activate disk error
Hi Karan,
Thank you for reply and help, to keep names simple let us use the installation
guide naming
http://ceph.com/docs/master/_images/ditaa-ab0a88be6a09668151342b36da8ceabaf0528f79.png
so I
Copy <cluster_name>.client.admin.keyring from ceph-node1 (Monitor node) to
/etc/ceph at ceph-node2 (1st OSD node) sudo scp
ceph-node1:/etc/ceph/ceph.client.admin.keyring ceph-node2:/etc/ceph/ And Copy
/var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-node1 (Monitor node) to
/var/lib/ceph/bootstrap-osd ceph-node2 (1st OSD node) sudo scp
ceph-server02:/var/lib/ceph/bootstrap-osd/ceph.keyring
ceph-server02:/var/lib/ceph/bootstrap-osd/
then using the Admin node (ceph-deploy) I run
[ceph@ceph-deploy my-cluster]$ ceph-deploy disk list ceph-node2
[ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy disk list
ceph-node2 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with
sudo [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final
[ceph_deploy.osd][DEBUG ] Listing disks on ceph-node2...
[ceph-server02][INFO ] Running command: ceph-disk list [ceph-server02][INFO ]
/dev/sda :
[ceph-server02][INFO ] /dev/sda1 other, ext4, mounted on /boot
[ceph-server02][INFO ] /dev/sda2 other, LVM2_member [ceph-server02][INFO ]
/dev/sdb :
[ceph-server02][INFO ] /dev/sdb1 ceph data, prepared, cluster ceph, journal
/dev/sdb2 [ceph-server02][INFO ] /dev/sdb2 ceph journal, for /dev/sdb1
[ceph-server02][INFO ] /dev/sr0 other, unknown
Then also using ceph Admin node (ceph-deply) I run
[ceph@ceph-deploy my-cluster]$ ceph-deploy osd activate ceph-node2:/dev/sdb1
[ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate
ceph-node2:/dev/sdb1 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
ceph-node2:/dev/sdb1:
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG
] activating host ceph-node2 disk /dev/sdb1 [ceph_deploy.osd][DEBUG ] will use
init type: sysvinit [ceph-server02][INFO ] Running command: ceph-disk-activate
--mark-init sysvinit --mount /dev/sdb1
Hang for 5 min
While hanging the logs at ceph-node2
[ceph@ceph-node2 ceph]$ ls -ltr
total 12
-rw-r--r-- 1 root root 0 Oct 30 03:44 ceph-osd..log
-rw-r--r-- 1 root root 0 Oct 30 03:44 ceph-osd.0.log
-rw-r--r-- 1 root root 0 Oct 30 03:44 ceph-client.admin.log
And logs at ceph-node1
[ceph@ceph-node1 ceph]$ ls -ltr
total 4520
-rw-r--r-- 1 root root 0 Oct 29 19:33 ceph-osd.ceph-server02.log
-rw-r--r-- 1 root root 0 Oct 30 03:13 ceph-osd..log
-rw-r--r-- 1 root root 0 Oct 30 03:13 ceph-osd.0.log
-rw------- 1 root root 0 Oct 30 03:13 ceph.log
-rw-r--r-- 1 root root 0 Oct 30 03:13 ceph-client.admin.log
-rw-r--r-- 1 root root 4415099 Oct 30 14:11 ceph-mon.ceph-server01.log
[ceph@ceph-node1 ceph]$ sudo tail ceph-mon.ceph-server01.log
2013-10-30 14:10:59.151566 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:10:59.151567 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:04.151701 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:04.151708 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:04.151744 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:04.151745 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:04.151755 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:04.151756 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:09.151908 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:09.151915 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:09.151948 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:09.151950 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:09.151959 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:09.151960 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:14.152094 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:14.152102 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:14.152138 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:14.152140 lease_expire=0.000000 has v0 lc 80
2013-10-30 14:11:14.152150 7f6a0c1a4700 1
mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable
now=2013-10-30 14:11:14.152150 lease_expire=0.000000 has v0 lc 80
Then ceph admin (ceph-deploy) node comes with error
[ceph-server02][ERROR ] 2013-10-30 10:52:03.307808 7f48a8169700 0 -- :/1002416
>> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1
c=0x7f4898000a60).fault [ceph-server02][ERROR ] 2013-10-30 10:52:06.308108
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48980008c0).fault [ceph-server02][ERROR ]
2013-10-30 10:52:09.308682 7f48a8169700 0 -- :/1002416 >>
192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1
c=0x7f48980130e0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:12.309033
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault [ceph-server02][ERROR ]
2013-10-30 10:52:15.309477 7f48a8169700 0 -- :/1002416 >>
192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1
c=0x7f48980130e0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:18.309909
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault [ceph-server02][ERROR ]
2013-10-30 10:52:21.310938 7f48a8169700 0 -- :/1002416 >>
192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1
c=0x7f4898002ef0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:24.310740
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48980037c0).fault [ceph-server02][ERROR ]
2013-10-30 10:52:27.311184 7f48a8169700 0 -- :/1002416 >>
192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1
c=0x7f4898002ef0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:30.265401
7f48a98fb700 0 monclient(hunting): authenticate timed out after 300
[ceph-server02][ERROR ] 2013-10-30 10:52:30.265482 7f48a98fb700 0 librados:
client.bootstrap-osd authentication error (110) Connection timed out
[ceph-server02][ERROR ] Error connecting to cluster: Error
[ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate
After error the logs @ node1(Monitor) & node2(OSD) doesn't change, only logs in
ceph-deploy show
less ceph.log | grep '2013-10-30' | more
2013-10-30 14:04:26,090 [ceph_deploy.cli][INFO ] Invoked (1.2.7):
/usr/bin/ceph-deploy disk list ceph-server02
2013-10-30 14:04:26,111 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote
connection with sudo
2013-10-30 14:04:26,818 [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final
2013-10-30 14:04:26,819 [ceph_deploy.osd][DEBUG ] Listing disks on
ceph-server02...
2013-10-30 14:04:26,819 [ceph-server02][INFO ] Running command: ceph-disk list
2013-10-30 14:04:27,807 [ceph-server02][INFO ] /dev/sda :
2013-10-30 14:04:27,808 [ceph-server02][INFO ] /dev/sda1 other, ext4, mounted
on /boot
2013-10-30 14:04:27,808 [ceph-server02][INFO ] /dev/sda2 other, LVM2_member
2013-10-30 14:04:27,808 [ceph-server02][INFO ] /dev/sdb :
2013-10-30 14:04:27,809 [ceph-server02][INFO ] /dev/sdb1 ceph data, prepared,
cluster ceph, journal /dev/sdb2
2013-10-30 14:04:27,809 [ceph-server02][INFO ] /dev/sdb2 ceph journal, for
/dev/sdb1
2013-10-30 14:04:27,809 [ceph-server02][INFO ] /dev/sr0 other, unknown
2013-10-30 14:06:15,558 [ceph_deploy.cli][INFO ] Invoked (1.2.7):
/usr/bin/ceph-deploy osd activate ceph-server02:/dev/sdb
2013-10-30 14:06:15,559 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
ceph-server02:/dev/sdb:
2013-10-30 14:06:15,560 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote
connection with sudo
2013-10-30 14:06:15,953 [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final
2013-10-30 14:06:15,954 [ceph_deploy.osd][DEBUG ] activating host ceph-server02
disk /dev/sdb
2013-10-30 14:06:15,954 [ceph_deploy.osd][DEBUG ] will use init type: sysvinit
2013-10-30 14:06:15,955 [ceph-server02][INFO ] Running command:
ceph-disk-activate --mark-init sysvinit --mount /dev/sdb
2013-10-30 14:06:16,615 [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to
activate
2013-10-30 14:06:23,805 [ceph_deploy.cli][INFO ] Invoked (1.2.7):
/usr/bin/ceph-deploy osd activate ceph-server02:/dev/sdb1
2013-10-30 14:06:23,806 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
ceph-server02:/dev/sdb1:
2013-10-30 14:06:23,806 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote
connection with sudo
2013-10-30 14:06:24,199 [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final
2013-10-30 14:06:24,200 [ceph_deploy.osd][DEBUG ] activating host ceph-server02
disk /dev/sdb1
2013-10-30 14:06:24,200 [ceph_deploy.osd][DEBUG ] will use init type: sysvinit
2013-10-30 14:06:24,200 [ceph-server02][INFO ] Running command:
ceph-disk-activate --mark-init sysvinit --mount /dev/sdb1
2013-10-30 14:11:25,115 [ceph-server02][ERROR ] 2013-10-30 10:47:30.266005
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f48a4024480 sd=9
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48a40246e0).fault
2013-10-30 14:11:25,115 [ceph-server02][ERROR ] 2013-10-30 10:47:33.266367
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898000c00 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898000e60).fault
2013-10-30 14:11:25,116 [ceph-server02][ERROR ] 2013-10-30 10:47:36.267392
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898003010 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898003270).fault
2013-10-30 14:11:25,116 [ceph-server02][ERROR ] 2013-10-30 10:47:39.267733
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898003850 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898003ab0).fault
2013-10-30 14:11:25,116 [ceph-server02][ERROR ] 2013-10-30 10:47:42.268123
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f48980025d0 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002830).fault
2013-10-30 14:11:25,139 [ceph-server02][ERROR ] 2013-10-30 10:51:42.305798
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898004430 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f489800c940).fault
2013-10-30 14:11:25,139 [ceph-server02][ERROR ] 2013-10-30 10:51:45.305590
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f489800df70).fault
2013-10-30 14:11:25,140 [ceph-server02][ERROR ] 2013-10-30 10:51:48.305889
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800c450 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f489800c6b0).fault
2013-10-30 14:11:25,140 [ceph-server02][ERROR ] 2013-10-30 10:51:51.306287
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f489800df70).fault
2013-10-30 14:11:25,141 [ceph-server02][ERROR ] 2013-10-30 10:51:54.307138
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f489800f100).fault
2013-10-30 14:11:25,141 [ceph-server02][ERROR ] 2013-10-30 10:51:57.307693
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f489800df70).fault
2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:00.307526
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48980008c0).fault
2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:03.307808
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898000a60).fault
2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:06.308108
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48980008c0).fault
2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:09.308682
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48980130e0).fault
2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:12.309033
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault
2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:15.309477
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48980130e0).fault
2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:18.309909
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault
2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:21.310938
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002ef0).fault
2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:24.310740
7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f48980037c0).fault
2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:27.311184
7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002ef0).fault
2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:30.265401
7f48a98fb700 0 monclient(hunting): authenticate timed out after 300
2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:30.265482
7f48a98fb700 0 librados: client.bootstrap-osd authentication error (110)
Connection timed out
2013-10-30 14:11:25,145 [ceph-server02][ERROR ] Error connecting to cluster:
Error
2013-10-30 14:11:25,145 [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to
activate
Note: in my setup and installation guide, the Admin node(ceph-deploy) is a
separate server other than ceph-node1(Monitor) and ceph-node2, ceph-node3(OSD),
and the admin node doesn't required ceph installation only ceph-deploy, is that
right ? also admin node (ceph-deploy) gather keys from ceph-node1 (monitor
node) only, right ?
Regards,
Nabil Naim
-----Original Message-----
From: Karan Singh [mailto:ksi...@csc.fi]
Sent: Wednesday, October 30, 2013 11:44 AM
To: Nabil Naim
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] activate disk error
Hey Nabil
Reinstallation would not be a solution , during my ceph installation i done 8
time reinstallation of ceph in just 3 days , and then realised its not a
solution.
Anyway , lets dig your problem if you like :-)
Your Logs says that some problem with connecting with cluster
s=1 pgs=0 cs=0 l=1 c=0x7f0da800f3d0).fault [ceph-server02][ERROR ] 2013-10-29
21:54:47.679151 7f0db997a700 0 monclient(hunting): authenticate timed out
after 300 [ceph-server02][ERROR ] 2013-10-29 21:54:47.679252 7f0db997a700 0
librados: client.bootstrap-osd authentication error (110) Connection timed out
[ceph-server02][ERROR ] Error connecting to cluster: Error
[ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate
This problem might be related to key rings , pleas try this
1) On ceph-deploy , cd to your ceph installatino directory usually /etc/ceph or
my-cluster if you have changed it.
2) scp <cluster_name>.c lient.admin.keyring ceph-server02:/etc/ceph ( or your
ceph-server02 installation directory )
3) scp /var/lib/ceph/bootstrap-osd/ceph.keyring
ceph-server02:/var/lib/ceph/bootstrap-osd ( create bootstrap-osd direcrory on
ceph-server02 if not there )
4) Again try to activate your OSD , it should work
PS : Check again for your server name , file name , and directory as the names
would be specific to your environment.
Here the plan is to move keyrings from MONITOR node to OSD node.
Regards
Karan Singh
System Specialist Storage | CSC IT centre for science Espoo Finland
karan.si...@csc.fi
----- Original Message -----
From: "Nabil Naim" <nabil_n...@linkdatacenter.net>
To: ceph-users@lists.ceph.com
Sent: Tuesday, 29 October, 2013 9:15:06 PM
Subject: Re: [ceph-users] activate disk error
Also the prepare step done successfully
[ceph@ceph-deploy my-cluster]$ ceph-deploy disk list ceph-server02
[ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy disk list
ceph-server02 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection
with sudo [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final
[ceph_deploy.osd][DEBUG ] Listing disks on ceph-server02...
[ceph-server02][INFO ] Running command: ceph-disk list [ceph-server02][INFO ]
/dev/sda :
[ceph-server02][INFO ] /dev/sda1 other, ext4, mounted on /boot
[ceph-server02][INFO ] /dev/sda2 other, LVM2_member [ceph-server02][INFO ]
/dev/sdb :
[ceph-server02][INFO ] /dev/sdb1 ceph data, prepared, cluster ceph, journal
/dev/sdb2 [ceph-server02][INFO ] /dev/sdb2 ceph journal, for /dev/sdb1
[ceph-server02][INFO ] /dev/sr0 other, unknown
Regards,
Nabil Naim
-----Original Message-----
From: Nabil Naim
Sent: Tuesday, October 29, 2013 9:07 PM
To: 'ceph-users@lists.ceph.com'
Subject: RE: [ceph-users] activate disk error
Nothing on the ceph-server02 log
ceph-deploy osd activate ceph-server02:/dev/sdb1
s=1 pgs=0 cs=0 l=1 c=0x7f0da8013a80).fault [ceph-server02][ERROR ] 2013-10-29
21:54:38.712639 7f0db81e8700 0 -- :/1002801 >> 192.168.115.91:6789/0
pipe(0x7f0da800b350 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f0da800f3d0).fault
[ceph-server02][ERROR ] 2013-10-29 21:54:42.712477 7f0db82e9700 0 -- :/1002801
>> 192.168.115.91:6789/0 pipe(0x7f0da80008c0 sd=10 :0 s=1 pgs=0 cs=0 l=1
c=0x7f0da8013a50).fault [ceph-server02][ERROR ] 2013-10-29 21:54:45.713387
7f0db81e8700 0 -- :/1002801 >> 192.168.115.91:6789/0 pipe(0x7f0da800b350 sd=10
:0 s=1 pgs=0 cs=0 l=1 c=0x7f0da800f3d0).fault [ceph-server02][ERROR ]
2013-10-29 21:54:47.679151 7f0db997a700 0 monclient(hunting): authenticate
timed out after 300 [ceph-server02][ERROR ] 2013-10-29 21:54:47.679252
7f0db997a700 0 librados: client.bootstrap-osd authentication error (110)
Connection timed out [ceph-server02][ERROR ] Error connecting to cluster: Error
[ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate
only way to bypass the error Is to fully installed ceph-server02 as a new
cluster and monitor and osd node :-((((
any advice ? :-(
Regards,
Nabil Naim
-----Original Message-----
From: Nabil Naim
Sent: Monday, October 28, 2013 6:27 PM
To: 'Sage Weil'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] activate disk error
Hi Sage,
Thank you for reply
I try to use implement CEPH following
http://ceph.com/docs/master/start/quick-ceph-deploy/
All my servers are VMware instances, all steps working fine unless
prepare/create OSD , I try ceph-deploy osd prepare ceph-node2:/tmp/osd0
ceph-node3:/tmp/osd1 and aslo I try to use extra HD ceph-deploy osd create
ceph-node2:/dev/sdb1 ceph-node3:/dev/sdb1 each time in ceph-deploy osd activate
I got the same error [root@ceph-deploy my-cluster]# ceph-deploy -v osd activate
ceph-server02:/dev/sdb1
it gives
ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy -v osd activate
ceph-server02:/dev/sdb [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
ceph-server02:/dev/sdb:
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo
[ceph_deploy.osd][INFO ] Distro info: CentOS 6.2 Final [ceph_deploy.osd][DEBUG
] activating host ceph-server02 disk /dev/sdb [ceph_deploy.osd][DEBUG ] will
use init type: sysvinit [ceph-server02][INFO ] Running command:
ceph-disk-activate --mark-init sysvinit --mount /dev/sdb [root@ceph-deploy
my-cluster]# ceph-deploy -v osd activate ceph-server02:/dev/sdb1
[ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy -v osd activate
ceph-server02:/dev/sdb1 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
ceph-server02:/dev/sdb1:
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo
[ceph_deploy.osd][INFO ] Distro info: CentOS 6.2 Final [ceph_deploy.osd][DEBUG
] activating host ceph-server02 disk /dev/sdb1 [ceph_deploy.osd][DEBUG ] will
use init type: sysvinit [ceph-server02][INFO ] Running command:
ceph-disk-activate --mark-init sysvinit --mount /dev/sdb1
And suspend for a while then
[ceph-server02][ERROR ] 2013-10-24 18:36:56.049060 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35c0020430 sd=9 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35c0020690).fault [ceph-server02][ERROR ] 2013-10-24 18:36:59.047638
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4000c00 sd=9 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4000e60).fault [ceph-server02][ERROR ] 2013-10-24
18:37:02.049738 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4003010 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003270).fault
[ceph-server02][ERROR ] 2013-10-24 18:37:05.049212 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4003850 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4003ab0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:08.049732
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40025d0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4002830).fault [ceph-server02][ERROR ] 2013-10-24
18:37:11.050150 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4002cf0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4002f50).fault
[ceph-server02][ERROR ] 2013-10-24 18:37:14.050596 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004110 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4004370).fault [ceph-server02][ERROR ] 2013-10-24 18:37:17.050835
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004900 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4004b60).fault [ceph-server02][ERROR ] 2013-10-24
18:37:20.051166 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4005240 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40054a0).fault
[ceph-server02][ERROR ] 2013-10-24 18:37:23.051520 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4005960 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4005bc0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:26.051803
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40093b0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4009610).fault [ceph-server02][ERROR ] 2013-10-24
18:37:29.052464 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4009a60 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4009cc0).fault
[ceph-server02][ERROR ] 2013-10-24 18:37:32.052918 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b400a320 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400a580).fault [ceph-server02][ERROR ] 2013-10-24 18:37:35.053331
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400ab60 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400adc0).fault [ceph-server02][ERROR ] 2013-10-24
18:37:38.053733 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4007350 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40075b0).fault
[ceph-server02][ERROR ] 2013-10-24 18:37:41.054145 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b400d230 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400d490).fault [ceph-server02][ERROR ] 2013-10-24 18:37:44.054592
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400dbc0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400de20).fault [ceph-server02][ERROR ] 2013-10-24
18:37:47.055107 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4006440 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40066a0).fault
[ceph-server02][ERROR ] 2013-10-24 18:37:50.055587 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4006c30 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4006e90).fault [ceph-server02][ERROR ] 2013-10-24 18:37:53.055885
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4007c70 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4007ed0).fault [ceph-server02][ERROR ] 2013-10-24
18:37:56.056305 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b40084e0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4008740).fault
[ceph-server02][ERROR ] 2013-10-24 18:37:59.056735 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4008cd0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400aff0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:02.057308
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400b4b0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400b710).fault [ceph-server02][ERROR ] 2013-10-24
18:38:05.057724 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b400bae0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400bd40).fault
[ceph-server02][ERROR ] 2013-10-24 18:38:08.058137 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b400c340 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400c5a0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:11.058621
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400cb30 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400cd90).fault [ceph-server02][ERROR ] 2013-10-24
18:38:14.059029 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b400f2f0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400f550).fault
[ceph-server02][ERROR ] 2013-10-24 18:38:17.059473 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b400e270 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400e4d0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:20.059899
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400fa70 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400fcd0).fault [ceph-server02][ERROR ] 2013-10-24
18:38:23.060269 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b400eb90 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400edf0).fault
[ceph-server02][ERROR ] 2013-10-24 18:38:26.060671 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b40110e0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4011340).fault [ceph-server02][ERROR ] 2013-10-24 18:38:29.061057
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40008c0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400ed60).fault [ceph-server02][ERROR ] 2013-10-24
18:38:32.061511 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40112b0).fault
[ceph-server02][ERROR ] 2013-10-24 18:38:35.061779 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b40008c0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4003640).fault [ceph-server02][ERROR ] 2013-10-24 18:38:38.062252
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4002000).fault [ceph-server02][ERROR ] 2013-10-24
18:38:41.062654 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4003640 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40038a0).fault
[ceph-server02][ERROR ] 2013-10-24 18:38:44.063195 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b40008c0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b40025d0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:47.063618
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4003640 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b40038a0).fault [ceph-server02][ERROR ] 2013-10-24
18:38:50.064078 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40008c0).fault
[ceph-server02][ERROR ] 2013-10-24 18:38:53.064514 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4003640).fault [ceph-server02][ERROR ] 2013-10-24 18:38:56.064950
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b40008c0).fault [ceph-server02][ERROR ] 2013-10-24
18:38:59.065660 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40090e0).fault
[ceph-server02][ERROR ] 2013-10-24 18:39:02.065752 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b40040a0).fault [ceph-server02][ERROR ] 2013-10-24 18:39:05.066009
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4011850).fault [ceph-server02][ERROR ] 2013-10-24
18:39:08.066380 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400a110).fault
[ceph-server02][ERROR ] 2013-10-24 18:39:11.066675 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400d090).fault [ceph-server02][ERROR ] 2013-10-24 18:39:14.066973
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400d9b0).fault [ceph-server02][ERROR ] 2013-10-24
18:39:17.067354 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4007280).fault
[ceph-server02][ERROR ] 2013-10-24 18:39:20.067692 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400d160).fault [ceph-server02][ERROR ] 2013-10-24 18:39:23.068050
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4007930).fault [ceph-server02][ERROR ] 2013-10-24
18:39:26.068397 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4008340).fault
[ceph-server02][ERROR ] 2013-10-24 18:39:29.068753 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4008e70).fault [ceph-server02][ERROR ] 2013-10-24 18:39:32.069143
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400db50).fault [ceph-server02][ERROR ] 2013-10-24
18:39:35.069420 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4008c60 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40089f0).fault
[ceph-server02][ERROR ] 2013-10-24 18:39:38.069956 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400b370).fault [ceph-server02][ERROR ] 2013-10-24 18:39:41.070217
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b40089f0).fault [ceph-server02][ERROR ] 2013-10-24
18:39:44.070467 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b370).fault
[ceph-server02][ERROR ] 2013-10-24 18:39:47.070721 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400cb90).fault [ceph-server02][ERROR ] 2013-10-24 18:39:50.071074
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400b370).fault [ceph-server02][ERROR ] 2013-10-24
18:39:53.071415 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400e920).fault
[ceph-server02][ERROR ] 2013-10-24 18:39:56.071697 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400fa70).fault [ceph-server02][ERROR ] 2013-10-24 18:39:59.072149
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400cb90).fault [ceph-server02][ERROR ] 2013-10-24
18:40:02.072619 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400e7d0).fault
[ceph-server02][ERROR ] 2013-10-24 18:40:05.072983 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4000c00).fault [ceph-server02][ERROR ] 2013-10-24 18:40:08.073397
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4001e50).fault [ceph-server02][ERROR ] 2013-10-24
18:40:11.073780 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003d60).fault
[ceph-server02][ERROR ] 2013-10-24 18:40:14.074099 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4003a10).fault [ceph-server02][ERROR ] 2013-10-24 18:40:17.074457
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4003d60).fault [ceph-server02][ERROR ] 2013-10-24
18:40:20.074818 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4004890).fault
[ceph-server02][ERROR ] 2013-10-24 18:40:23.075174 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4003d60).fault [ceph-server02][ERROR ] 2013-10-24 18:40:26.075475
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4004890).fault [ceph-server02][ERROR ] 2013-10-24
18:40:29.075947 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40058f0).fault
[ceph-server02][ERROR ] 2013-10-24 18:40:32.076433 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4002c90).fault [ceph-server02][ERROR ] 2013-10-24 18:40:35.076933
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b40090e0).fault [ceph-server02][ERROR ] 2013-10-24
18:40:38.077319 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4010210 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4010470).fault
[ceph-server02][ERROR ] 2013-10-24 18:40:41.077636 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b40040a0).fault [ceph-server02][ERROR ] 2013-10-24 18:40:44.078049
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4010210 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4010470).fault [ceph-server02][ERROR ] 2013-10-24
18:40:47.078406 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40040a0).fault
[ceph-server02][ERROR ] 2013-10-24 18:40:50.078961 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4010210 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4010470).fault [ceph-server02][ERROR ] 2013-10-24 18:40:53.079379
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4006510).fault [ceph-server02][ERROR ] 2013-10-24
18:40:56.079689 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4010000 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4010260).fault
[ceph-server02][ERROR ] 2013-10-24 18:40:59.080329 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b4006510).fault [ceph-server02][ERROR ] 2013-10-24 18:41:02.080887
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4006bc0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b4007c30).fault [ceph-server02][ERROR ] 2013-10-24
18:41:05.081282 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b940).fault
[ceph-server02][ERROR ] 2013-10-24 18:41:08.081745 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4006bc0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400b590).fault [ceph-server02][ERROR ] 2013-10-24 18:41:11.082121
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400b940).fault [ceph-server02][ERROR ] 2013-10-24
18:41:14.082484 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4006bc0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40099e0).fault
[ceph-server02][ERROR ] 2013-10-24 18:41:17.082768 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400bbb0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:20.083182
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40083f0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400f8d0).fault [ceph-server02][ERROR ] 2013-10-24
18:41:23.083607 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400fc10).fault
[ceph-server02][ERROR ] 2013-10-24 18:41:26.083897 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b40083f0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400f8d0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:29.084225
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b40015f0).fault [ceph-server02][ERROR ] 2013-10-24
18:41:32.084656 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b40083f0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400eb50).fault
[ceph-server02][ERROR ] 2013-10-24 18:41:35.085102 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b40015f0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:38.085421
7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40083f0 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400eb50).fault [ceph-server02][ERROR ] 2013-10-24
18:41:41.085742 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400efa0).fault
[ceph-server02][ERROR ] 2013-10-24 18:41:44.086157 7f35c4986700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4006bc0 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400eb50).fault [ceph-server02][ERROR ] 2013-10-24 18:41:47.086679
7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1
pgs=0 cs=0 l=1 c=0x7f35b400efa0).fault [ceph-server02][ERROR ] 2013-10-24
18:41:50.087077 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0
pipe(0x7f35b400ff80 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400eb50).fault
[ceph-server02][ERROR ] 2013-10-24 18:41:53.087514 7f35c4885700 0 -- :/1006405
>> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1
c=0x7f35b400efa0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:56.046946
7f35c8058700 0 monclient(hunting): authenticate timed out after 300
[ceph-server02][ERROR ] 2013-10-24 18:41:56.047026 7f35c8058700 0 librados:
client.bootstrap-osd authentication error (110) Connection timed out
[ceph-server02][ERROR ] Error connecting to cluster: Error
[ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate
Note: I try as tutorial to use ceph user and root user as well with no success
-----Original Message-----
From: Sage Weil [mailto:s...@inktank.com]
Sent: Monday, October 28, 2013 6:10 PM
To: Nabil Naim
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] activate disk error
On Mon, 28 Oct 2013, Nabil Naim wrote:
>
> Any one have clue why this error happen
>
> 2013-10-28 14:12:23.817719 7fe95437a700 0 -- :/1008986 >>
> 192.168.115.91:6789/0 pipe(0x7fe944010d00 sd=5 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fe9440046b0).fault
>
> When I try to activate disk
It looks like 192.168.115.91 is one of your monitors and it was either down or
there was a transient tcp connection problem.
sage
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
