You have saved my day! Thank you so much :) no seems to be working not sure why it has that behaivor... Thank you so much!
Jesus Chavez SYSTEMS ENGINEER-C.SALES jesch...@cisco.com<mailto:jesch...@cisco.com> Phone: +52 55 5267 3146<tel:+52%2055%205267%203146> Mobile: +51 1 5538883255<tel:+51%201%205538883255> CCIE - 44433 On Apr 14, 2015, at 12:34 AM, "oyym...@gmail.com<mailto:oyym...@gmail.com>" <oyym...@gmail.com<mailto:oyym...@gmail.com>> wrote: UUID strings in the file /etc/fstab refer to devices. you can retrieve UUID using command blkid, for instance: # blkid /dev/sda1 /dev/sda1: UUID="1bcb1cbb-abd2-4cfa-bf89-1726ea6cd2fa" TYPE="xfs" ________________________________ oyym...@gmail.com<mailto:oyym...@gmail.com> From: Jesus Chavez (jeschave)<mailto:jesch...@cisco.com> Date: 2015-04-14 07:03 To: oyym...@gmail.com<mailto:oyym...@gmail.com> CC: ceph-users<mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] ERROR: missing keyring, cannot use cephx for authentication How did you get the UUIDs without mounting the osds? Thanks Jesus Chavez SYSTEMS ENGINEER-C.SALES jesch...@cisco.com<mailto:jesch...@cisco.com> Phone: +52 55 5267 3146<tel:+52%2055%205267%203146> Mobile: +51 1 5538883255<tel:+51%201%205538883255> CCIE - 44433 On Apr 10, 2015, at 11:47 PM, "oyym...@gmail.com<mailto:oyym...@gmail.com>" <oyym...@gmail.com<mailto:oyym...@gmail.com>> wrote: In my cluster, mount osd devices manually in /etc/fstab, for instances: vi /etc/fstab # ... UUID=98c87892-21cd-481b-9619-1a4a65620d79 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,inode64,logbsize=256k,delaylog 0 3 UUID=d56aeb7b-6160-4872-91dc-346b39019081 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,inode64,logbsize=256k,delaylog 0 3 UUID=db9782a6-b661-46b5-b993-a4d96bdc37d8 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,inode64,logbsize=256k,delaylog 0 3 UUID=ec95aa19-127d-410b-86a3-d68ee6baff2e /var/lib/ceph/osd/ceph-3 xfs rw,noatime,inode64,logbsize=256k,delaylog 0 3 UUID=44694045-d9a3-4df4-948c-a77696d22406 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,inode64,logbsize=256k,delaylog 0 3 mount -a mount /dev/sda12 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime,seclabel,attr2,inode64,logbsize=256k,noquota) /dev/sda13 on /var/lib/ceph/osd/ceph-1 type xfs (rw,noatime,seclabel,attr2,inode64,logbsize=256k,noquota) /dev/sdb1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,noatime,seclabel,attr2,inode64,logbsize=256k,noquota) /dev/sdb2 on /var/lib/ceph/osd/ceph-3 type xfs (rw,noatime,seclabel,attr2,inode64,logbsize=256k,noquota) /dev/sdb3 on /var/lib/ceph/osd/ceph-4 type xfs (rw,noatime,seclabel,attr2,inode64,logbsize=256k,noquota) reboot ceph osd tree # id weight type name up/down reweight -1 22.75 root default -2 4.55 host storage1 0 0.91 osd.0 up 1 1 0.91 osd.1 up 1 2 0.91 osd.2 up 1 3 0.91 osd.3 up 1 4 0.91 osd.4 up 1 -3 4.55 host storage2 5 0.91 osd.5 up 1 6 0.91 osd.6 up 1 7 0.91 osd.7 up 1 8 0.91 osd.8 up 1 9 0.91 osd.9 up 1 -4 4.55 host lkl-storage3 10 0.91 osd.10 up 1 11 0.91 osd.11 up 1 12 0.91 osd.12 up 1 13 0.91 osd.13 up 1 14 0.91 osd.14 up 1 -5 4.55 host storage4 15 0.91 osd.15 up 1 16 0.91 osd.16 up 1 17 0.91 osd.17 up 1 18 0.91 osd.18 up 1 19 0.91 osd.19 up 1 -6 4.55 host storage5 20 0.91 osd.20 up 1 21 0.91 osd.21 up 1 22 0.91 osd.22 up 1 23 0.91 osd.23 up 1 24 0.91 osd.24 up 1 ________________________________ oyym...@gmail.com<mailto:oyym...@gmail.com> From: Jesus Chavez (jeschave)<mailto:jesch...@cisco.com> Date: 2015-04-10 22:17 To: oyym...@gmail.com<mailto:oyym...@gmail.com> Subject: Re: [ceph-users] ERROR: missing keyring, cannot use cephx for authentication Hi there, just to be sure if you could figure how to fix this problem? Thanks <image005(04-11-12-39-14).png> Jesus Chavez SYSTEMS ENGINEER-C.SALES jesch...@cisco.com<mailto:jesch...@cisco.com> Phone: +52 55 5267 3146 Mobile: +51 1 5538883255 CCIE - 44433 Cisco.com<http://www.cisco.com/> <image006(04-11-12-39-14).gif> Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. On Mar 27, 2015, at 8:05 AM, Jesus Chavez (jeschave) <jesch...@cisco.com<mailto:jesch...@cisco.com>> wrote: Did you know how to solve it? it is really weird… Thanks! <image005.png> Jesus Chavez SYSTEMS ENGINEER-C.SALES jesch...@cisco.com<mailto:jesch...@cisco.com> Phone: +52 55 5267 3146 Mobile: +51 1 5538883255 CCIE - 44433 Cisco.com<http://www.cisco.com/> <image006.gif> Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. On Mar 25, 2015, at 1:31 AM, oyym...@gmail.com<mailto:oyym...@gmail.com> wrote: Hi，Jesus I encountered similar problem. 1. shut down one of nodes， but all osds can't reactive on the node after reboot. 2. run service ceph restart manually, got the same error message: [root@storage4 ~]# /etc/init.d/ceph start === osd.15 === 2015-03-23 14:43:32.399811 7fed0fcf4700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2015-03-23 14:43:32.399814 7fed0fcf4700 0 librados: osd.15 initialization error (2) No such file or directory Error connecting to cluster: ObjectNotFound failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.15 --keyring=/var/lib/ceph/osd/ceph-15/keyring osd crush create-or-move -- 15 0.19 host=storage4 root=default' ...... 3. ll /var/lib/ceph/osd/ceph-15/ total 0 all files disappeared in the /var/lib/ceph/osd/ceph-15/ ________________________________ oyym...@gmail.com<mailto:oyym...@gmail.com> From: Jesus Chavez (jeschave)<mailto:jesch...@cisco.com> Date: 2015-03-24 05:09 To: ceph-users<mailto:ceph-users@lists.ceph.com> Subject: [ceph-users] ERROR: missing keyring, cannot use cephx for authentication Hi all, I did HA failover test shutting down 1 node and I see that only 1 OSD came up after reboot: [root@geminis ceph]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 50G 4.5G 46G 9% / devtmpfs 126G 0 126G 0% /dev tmpfs 126G 80K 126G 1% /dev/shm tmpfs 126G 9.9M 126G 1% /run tmpfs 126G 0 126G 0% /sys/fs/cgroup /dev/sda1 494M 165M 330M 34% /boot /dev/mapper/rhel-home 36G 44M 36G 1% /home /dev/sdc1 3.7T 134M 3.7T 1% /var/lib/ceph/osd/ceph-14 If I run service ceph restart I got this error message… Stopping Ceph osd.94 on geminis...done === osd.94 === 2015-03-23 15:05:41.632505 7fe7b9941700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2015-03-23 15:05:41.632508 7fe7b9941700 0 librados: osd.94 initialization error (2) No such file or directory Error connecting to cluster: ObjectNotFound failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.94 --keyring=/var/lib/ceph/osd/ceph-94/keyring osd crush create-or-move -- 94 0.05 host=geminis root=default I have ceph.conf and ceph.client.admin.keyring under /etc/ceph: [root@geminis ceph]# ls /etc/ceph ceph.client.admin.keyring ceph.conf rbdmap tmp1OqNFi tmptQ0a1P [root@geminis ceph]# does anybody know what could be wrong? Thanks <image005(03-25-14-50-03).png> Jesus Chavez SYSTEMS ENGINEER-C.SALES jesch...@cisco.com<mailto:jesch...@cisco.com> Phone: +52 55 5267 3146 Mobile: +51 1 5538883255 CCIE - 44433 Cisco.com<http://www.cisco.com/> <image006(03-25-14-50-03).gif> Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
