Hello, This is an interesting topic and would like to know a solution to this problem. Does that mean we should never use Dell storage as ceph storage device? I have similar setup with Dell 4 iscsi LUNs attached to openstack controller and compute node in active-active situation.
As they were in active active 1 selected first 2 luns as osd on node1 and last 2 as osd on node 2. Is it ok to have this configuration specially when and node will be down or considering live migration. Regards Gaurav Goyal On 10-Jul-2016 9:02 pm, "Christian Balzer" <ch...@gol.com> wrote: > > Hello, > > On Sun, 10 Jul 2016 12:46:39 +0000 (UTC) William Josefsson wrote: > > > Hi everyone, > > > > I have problem with swapping drive and partition names on reboot. My > > Ceph is Hammer on CentOS7, Dell R730 6xSSD (2xSSD OS RAID1 PERC, > > 4xSSD=Journal drives), 18x1.8T SAS for OSDs. > > > > Whenever I reboot, drives randomly seem to change names. This is > > extremely dangerous and frustrating when I've initially setup CEPH with > > ceph-deploy, zap, prepare and activate. It has happened that I've > > accidentally erased wrong disk too when e.g. /dev/sdX had > > become /dev/sdY. > > > This isn't a Ceph specific question per se and you could probably keep > things from moving around by enforcing module loads in a particular order. > > But that of course still wouldn't help if something else changed or a > drive totally failed. > > So in the context of Ceph, it doesn't (shouldn't) care if the OSD (HDD) > changes names, especially since you did set it up with ceph-deploy. > > And to avoid the journals getting jumbled up, do what everybody does > (outside of Ceph as well), use /dev/disk/by-id or uuid. > > Like: > --- > # ls -la /var/lib/ceph/osd/ceph-28/ > > journal -> /dev/disk/by-id/wwn-0x55cd2e404b73d569-part3 > --- > > Christian > > Please see an output below of how this drive swapping below appears SDC > > is shifted, indexes and drive names got shuffled. Ceph OSDs didn't come > > up properly. > > > > Please advice on how to get this corrected, with no more drive name > > shuffling. Can this be due to the PERC HW raid? thx will > > > > > > > > POST REBOOT 2 (expected outcome.. with sda,sdb,sdc,sdd as journal. sdw > > is a perc raid1) > > > > > > [cephnode3][INFO ] Running command: sudo /usr/sbin/ceph-disk list > > [cephnode3][DEBUG ] /dev/sda : > > [cephnode3][DEBUG ] /dev/sda1 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda2 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sda3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sda4 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda5 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdb : [cephnode3][DEBUG ] /dev/sdb1 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb2 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdb3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sdb4 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb5 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdc : [cephnode3][DEBUG ] /dev/sdc1 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdc2 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdc3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sdc4 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdc5 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdd : [cephnode3][DEBUG ] /dev/sdd1 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd2 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdd3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sdd4 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd5 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sde : [cephnode3][DEBUG ] /dev/sde1 ceph data, active, cluster > > ceph, osd.0 [cephnode3][DEBUG ] /dev/sdf : [cephnode3][DEBUG > > ] /dev/sdf1 ceph data, active, cluster ceph, osd.1 [cephnode3][DEBUG > > ] /dev/sdg : [cephnode3][DEBUG ] /dev/sdg1 ceph data, active, cluster > > ceph, osd.2 [cephnode3][DEBUG ] /dev/sdh : [cephnode3][DEBUG > > ] /dev/sdh1 ceph data, active, cluster ceph, osd.3 [cephnode3][DEBUG > > ] /dev/sdi : [cephnode3][DEBUG ] /dev/sdi1 ceph data, active, cluster > > ceph, osd.4 [cephnode3][DEBUG ] /dev/sdj : [cephnode3][DEBUG > > ] /dev/sdj1 ceph data, active, cluster ceph, osd.5 [cephnode3][DEBUG > > ] /dev/sdk : [cephnode3][DEBUG ] /dev/sdk1 ceph data, active, cluster > > ceph, osd.6 [cephnode3][DEBUG ] /dev/sdl : [cephnode3][DEBUG > > ] /dev/sdl1 ceph data, active, cluster ceph, osd.7 [cephnode3][DEBUG > > ] /dev/sdm : [cephnode3][DEBUG ] /dev/sdm1 other, xfs > > [cephnode3][DEBUG ] /dev/sdn : > > [cephnode3][DEBUG ] /dev/sdn1 ceph data, active, cluster ceph, osd.9 > > [cephnode3][DEBUG ] /dev/sdo : > > [cephnode3][DEBUG ] /dev/sdo1 ceph data, active, cluster ceph, osd.10 > > [cephnode3][DEBUG ] /dev/sdp : > > [cephnode3][DEBUG ] /dev/sdp1 ceph data, active, cluster ceph, osd.11 > > [cephnode3][DEBUG ] /dev/sdq : > > [cephnode3][DEBUG ] /dev/sdq1 ceph data, active, cluster ceph, osd.12 > > [cephnode3][DEBUG ] /dev/sdr : > > [cephnode3][DEBUG ] /dev/sdr1 ceph data, active, cluster ceph, osd.13 > > [cephnode3][DEBUG ] /dev/sds : > > [cephnode3][DEBUG ] /dev/sds1 ceph data, active, cluster ceph, osd.14 > > [cephnode3][DEBUG ] /dev/sdt : > > [cephnode3][DEBUG ] /dev/sdt1 ceph data, active, cluster ceph, osd.15 > > [cephnode3][DEBUG ] /dev/sdu : > > [cephnode3][DEBUG ] /dev/sdu1 ceph data, active, cluster ceph, osd.16 > > [cephnode3][DEBUG ] /dev/sdv : > > [cephnode3][DEBUG ] /dev/sdv1 ceph data, active, cluster ceph, osd.17 > > [cephnode3][DEBUG ] /dev/sdw : > > [cephnode3][DEBUG ] /dev/sdw1 other, xfs, mounted on / > > [cephnode3][DEBUG ] /dev/sdw2 swap, swap > > > > > > POST REBOOT 1: > > > > > > [cephnode3][DEBUG ] /dev/sda : > > [cephnode3][DEBUG ] /dev/sda1 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda2 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sda3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sda4 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda5 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdb : [cephnode3][DEBUG ] /dev/sdb1 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb2 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdb3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sdb4 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb5 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdc : [cephnode3][DEBUG ] /dev/sdc1 ceph data, active, cluster > > ceph, osd.13 [cephnode3][DEBUG ] /dev/sdd : [cephnode3][DEBUG > > ] /dev/sdd1 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sdd2 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd3 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdd4 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sdd5 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde : > > [cephnode3][DEBUG ] /dev/sde1 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde2 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sde3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > > [cephnode3][DEBUG ] /dev/sde4 other, > > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde5 > > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > > ] /dev/sdf : [cephnode3][DEBUG ] /dev/sdf1 ceph data, active, cluster > > ceph, osd.0 [cephnode3][DEBUG ] /dev/sdg : [cephnode3][DEBUG > > ] /dev/sdg1 ceph data, active, cluster ceph, osd.14 [cephnode3][DEBUG > > ] /dev/sdh : [cephnode3][DEBUG ] /dev/sdh1 ceph data, active, cluster > > ceph, osd.1 [cephnode3][DEBUG ] /dev/sdi : [cephnode3][DEBUG > > ] /dev/sdi1 ceph data, active, cluster ceph, osd.2 [cephnode3][DEBUG > > ] /dev/sdj : [cephnode3][DEBUG ] /dev/sdj1 ceph data, active, cluster > > ceph, osd.3 [cephnode3][DEBUG ] /dev/sdk : [cephnode3][DEBUG > > ] /dev/sdk1 ceph data, active, cluster ceph, osd.4 [cephnode3][DEBUG > > ] /dev/sdl : [cephnode3][DEBUG ] /dev/sdl1 ceph data, active, cluster > > ceph, osd.15 [cephnode3][DEBUG ] /dev/sdm : [cephnode3][DEBUG > > ] /dev/sdm1 ceph data, active, cluster ceph, osd.5 [cephnode3][DEBUG > > ] /dev/sdn : [cephnode3][DEBUG ] /dev/sdn1 ceph data, active, cluster > > ceph, osd.6 [cephnode3][DEBUG ] /dev/sdo : > > [cephnode3][DEBUG ] /dev/sdo1 ceph data, active, cluster ceph, osd.7 > > [cephnode3][DEBUG ] /dev/sdp : > > [cephnode3][DEBUG ] /dev/sdp1 other, xfs > > [cephnode3][DEBUG ] /dev/sdq : > > [cephnode3][DEBUG ] /dev/sdq1 ceph data, active, cluster ceph, osd.9 > > [cephnode3][DEBUG ] /dev/sdr : > > [cephnode3][DEBUG ] /dev/sdr1 ceph data, active, cluster ceph, osd.16 > > [cephnode3][DEBUG ] /dev/sds : > > [cephnode3][DEBUG ] /dev/sds1 ceph data, active, cluster ceph, osd.10 > > [cephnode3][DEBUG ] /dev/sdt : > > [cephnode3][DEBUG ] /dev/sdt1 ceph data, active, cluster ceph, osd.11 > > [cephnode3][DEBUG ] /dev/sdu : > > [cephnode3][DEBUG ] /dev/sdu1 ceph data, active, cluster ceph, osd.12 > > [cephnode3][DEBUG ] /dev/sdv : > > [cephnode3][DEBUG ] /dev/sdv1 ceph data, active, cluster ceph, osd.17 > > [cephnode3][DEBUG ] /dev/sdw : > > [cephnode3][DEBUG ] /dev/sdw1 other, xfs, mounted on / > > [cephnode3][DEBUG ] /dev/sdw2 swap, swap > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Global OnLine Japan/Rakuten Communications > http://www.gol.com/ > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
