I had a similar issue when upgrading. When I originally created by journal partitions, I never set the partition type GUID to the Ceph Journal GUID (https://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_type_GUIDs). Instead it was set as "basic data partition". Pre-Infernalis this wasn't a problem since the daemons ran as root.
Found this bug http://tracker.ceph.com/issues/12877 and noticed that ceph does the ownership udev changes if the partition has a Ceph GUID. Setting it manually on the journal partitions as follows fixed the issue (in my system journals are sda1-5, sdb1-5, sdc1-5): for i in a b c;do for j in {1..5};do sgdisk -t ${j}:45B0969E-9B03-4F30-B4C6-B4B80CEFF106 /dev/sd${i};done;done This seems like a more robust method of fixing this issue than by setting additional udev rules based on the current device name. In the past, I've had block devices remap to /dev/sdX on reboot, which would break under the custom udev rules proposed in this thread. This might also be worth adding to the upgrade docs. -Steve On 11/19/2015 08:46 AM, David Riedl wrote: > Thanks again! It works now. > But now I have another problem. > > The daemons are working now, even after a restart. But the OSDs won't > talk to the rest of the cluster. > > osdmap e5058: 12 osds: 8 up, 8 in; > > The command > # ceph osd in osd.1 > tells me > marked in osd.1. > > # ceph status > tells me > 1/9 in osds are down > but that disappears after a while. > > Right now I have mixed OSDs from infernalis and the latest hammer > release. (that may be the most crucial information and even the cause > of the problem), but I am not sure. > > Sorry to bother you but this is my second day with these problems and > it's nerve wrecking. > > Regards > > David > > On 19.11.2015 14:29, Mykola Dvornik wrote: >> I am also using centos7.x. /usr/lib/udev/rules.d/ should be fine. If >> not, one can always symlink to /etc/udev/rules.d/. >> >> On 19 November 2015 at 14:13, David Riedl <david.ri...@wingcon.com >> <mailto:david.ri...@wingcon.com>> wrote: >> >> Thanks for the fix! >> Two questions though: >> Is that the right place for the udev rule? I have CentOS 7. The >> folder exists, but all the other udev rules are in >> /usr/lib/udev/rules.d/. >> Can I just create a new file named "89-ceph-journal.rules" in >> the /usr/lib/udev/rules.d/ folder? >> >> >> Regards >> >> David >> >> >> On 19.11.2015 14:02, Mykola Dvornik wrote: >>> cat /etc/udev/rules.d/89-ceph-journal.rules >>> >>> KERNEL=="sdd?" SUBSYSTEM=="block" OWNER="ceph" GROUP="disk" >>> MODE="0660" >>> >>> On 19 November 2015 at 13:54, Mykola <mykola.dvor...@gmail.com> >>> wrote: >>> >>> I am afraid one would need an udev rule to make it persistent. >>> >>> >>> >>> Sent from Outlook Mail >>> <http://go.microsoft.com/fwlink/?LinkId=550987> for Windows >>> 10 phone >>> >>> >>> >>> >>> *From: *David Riedl <mailto:david.ri...@wingcon.com> >>> *Sent: *Thursday, November 19, 2015 1:42 PM >>> *To: *ceph-us...@ceph.com <mailto:ceph-us...@ceph.com> >>> *Subject: *Re: [ceph-users] Can't activate osd in infernalis >>> >>> >>> >>> I fixed the issue and opened a ticket on the ceph-deploy bug >>> tracker >>> >>> http://tracker.ceph.com/issues/13833 >>> >>> >>> >>> tl;dr: >>> >>> change permission of the ssd journal partition with >>> >>> chown ceph:ceph /dev/sdd1 >>> >>> >>> >>> On 19.11.2015 11:38, David Riedl wrote: >>> >>> > Hi everyone. >>> >>> > I updated one of my hammer osd nodes to infernalis today. >>> >>> > After many problems with the upgrading process of the >>> running OSDs, I >>> >>> > decided to wipe them and start anew. >>> >>> > I reinstalled all packages and deleted all partitions on >>> the OSDs and >>> >>> > the SSD journal drive. >>> >>> > I zapped the disks with ceph-deploy and also prepared them >>> with >>> >>> > ceph-deploy. >>> >>> > Selinux state is enabled (disabling it didn't help though). >>> >>> > >>> >>> > After executing "ceph-deploy osd activate >>> ceph01:/dev/sda1:/dev/sdd1" >>> >>> > I get the following error message from ceph-deploy: >>> >>> > >>> >>> > >>> >>> > [ceph01][WARNIN] INFO:ceph-disk:Running command: >>> /usr/bin/ceph >>> >>> > --cluster ceph --name client.bootstrap-osd --keyring >>> >>> > /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o >>> >>> > /var/lib/ceph/tmp/mnt.pmHRuu/activate.monmap >>> >>> > [ceph01][WARNIN] 2015-11-19 11:22:53.974765 7f1a06852700 >>> 0 -- >>> >>> > :/3225863658 >> 10.20.60.10:6789/0 >>> <http://10.20.60.10:6789/0> pipe(0x7f19f8062590 sd=4 :0 s=1 >>> >>> > pgs=0 cs=0 l=1 c=0x7f19f805c1b0).fault >>> >>> > [ceph01][WARNIN] got monmap epoch 16 >>> >>> > [ceph01][WARNIN] INFO:ceph-disk:Running command: >>> /usr/bin/ceph-osd >>> >>> > --cluster ceph --mkfs --mkkey -i 0 --monmap >>> >>> > /var/lib/ceph/tmp/mnt.pmHRuu/activate.monmap --osd-data >>> >>> > /var/lib/ceph/tmp/mnt.pmHRuu --osd-journal >>> >>> > /var/lib/ceph/tmp/mnt.pmHRuu/journal --osd-uuid >>> >>> > de162e24-16b6-4796-b6b9-774fdb8ec234 --keyring >>> >>> > /var/lib/ceph/tmp/mnt.pmHRuu/keyring --setuser ceph >>> --setgroup ceph >>> >>> > [ceph01][WARNIN] 2015-11-19 11:22:57.237096 7fb458bb7900 -1 >>> >>> > filestore(/var/lib/ceph/tmp/mnt.pmHRuu) mkjournal error >>> creating >>> >>> > journal on /var/lib/ceph/tmp/mnt.pmHRuu/journal: (13) >>> Permission denied >>> >>> > [ceph01][WARNIN] 2015-11-19 11:22:57.237118 7fb458bb7900 >>> -1 OSD::mkfs: >>> >>> > ObjectStore::mkfs failed with error -13 >>> >>> > [ceph01][WARNIN] 2015-11-19 11:22:57.237157 7fb458bb7900 >>> -1 ** ERROR: >>> >>> > error creating empty object store in >>> /var/lib/ceph/tmp/mnt.pmHRuu: >>> >>> > (13) Permission denied >>> >>> > [ceph01][WARNIN] ERROR:ceph-disk:Failed to activate >>> >>> > [ceph01][WARNIN] DEBUG:ceph-disk:Unmounting >>> /var/lib/ceph/tmp/mnt.pmHRuu >>> >>> > [ceph01][WARNIN] INFO:ceph-disk:Running command: >>> /bin/umount -- >>> >>> > /var/lib/ceph/tmp/mnt.pmHRuu >>> >>> > [ceph01][WARNIN] Traceback (most recent call last): >>> >>> > [ceph01][WARNIN] File "/usr/sbin/ceph-disk", line 3576, >>> in <module> >>> >>> > [ceph01][WARNIN] main(sys.argv[1:]) >>> >>> > [ceph01][WARNIN] File "/usr/sbin/ceph-disk", line 3530, >>> in main >>> >>> > [ceph01][WARNIN] args.func(args) >>> >>> > [ceph01][WARNIN] File "/usr/sbin/ceph-disk", line 2424, in >>> >>> > main_activate >>> >>> > [ceph01][WARNIN] dmcrypt_key_dir=args.dmcrypt_key_dir, >>> >>> > [ceph01][WARNIN] File "/usr/sbin/ceph-disk", line 2197, in >>> >>> > mount_activate >>> >>> > [ceph01][WARNIN] (osd_id, cluster) = activate(path, >>> >>> > activate_key_template, init) >>> >>> > [ceph01][WARNIN] File "/usr/sbin/ceph-disk", line 2360, >>> in activate >>> >>> > [ceph01][WARNIN] keyring=keyring, >>> >>> > [ceph01][WARNIN] File "/usr/sbin/ceph-disk", line 1950, >>> in mkfs >>> >>> > [ceph01][WARNIN] '--setgroup', get_ceph_user(), >>> >>> > [ceph01][WARNIN] File "/usr/sbin/ceph-disk", line 349, in >>> >>> > command_check_call >>> >>> > [ceph01][WARNIN] return subprocess.check_call(arguments) >>> >>> > [ceph01][WARNIN] File >>> "/usr/lib64/python2.7/subprocess.py", line >>> >>> > 542, in check_call >>> >>> > [ceph01][WARNIN] raise CalledProcessError(retcode, cmd) >>> >>> > [ceph01][WARNIN] subprocess.CalledProcessError: Command >>> >>> > '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', >>> '--mkkey', '-i', >>> >>> > '0', '--monmap', >>> '/var/lib/ceph/tmp/mnt.pmHRuu/activate.monmap', >>> >>> > '--osd-data', '/var/lib/ceph/tmp/mnt.pmHRuu', >>> '--osd-journal', >>> >>> > '/var/lib/ceph/tmp/mnt.pmHRuu/journal', '--osd-uuid', >>> >>> > 'de162e24-16b6-4796-b6b9-774fdb8ec234', '--keyring', >>> >>> > '/var/lib/ceph/tmp/mnt.pmHRuu/keyring', '--setuser', 'ceph', >>> >>> > '--setgroup', 'ceph']' returned non-zero exit status 1 >>> >>> > [ceph01][ERROR ] RuntimeError: command returned non-zero >>> exit status: 1 >>> >>> > [ceph_deploy][ERROR ] RuntimeError: Failed to execute >>> command: >>> >>> > ceph-disk -v activate --mark-init systemd --mount /dev/sda1 >>> >>> > >>> >>> > The output of ls -lahn in /var/lib/ceph/ is >>> >>> > >>> >>> > drwxr-x---. 9 167 167 4,0K 19. Nov 10:32 . >>> >>> > drwxr-xr-x. 28 0 0 4,0K 19. Nov 11:14 .. >>> >>> > drwxr-x---. 2 167 167 6 10. Nov 13:06 bootstrap-mds >>> >>> > drwxr-x---. 2 167 167 25 19. Nov 10:48 bootstrap-osd >>> >>> > drwxr-x---. 2 167 167 6 10. Nov 13:06 bootstrap-rgw >>> >>> > drwxr-x---. 2 167 167 6 10. Nov 13:06 mds >>> >>> > drwxr-x---. 2 167 167 6 10. Nov 13:06 mon >>> >>> > drwxr-x---. 2 167 167 6 10. Nov 13:06 osd >>> >>> > drwxr-x---. 2 167 167 65 19. Nov 11:22 tmp >>> >>> > >>> >>> > >>> >>> > I hope someone can help me, I am really lost right now. >>> >>> > >>> >>> >>> >>> -- >>> >>> Mit freundlichen Grüßen >>> >>> >>> >>> David Riedl >>> >>> >>> >>> >>> >>> >>> >>> WINGcon GmbH Wireless New Generation - Consulting & Solutions >>> >>> >>> >>> Phone: +49 (0) 7543 9661 - 26 >>> <tel:%2B49%20%280%29%207543%209661%20-%2026> >>> >>> E-Mail: david.ri...@wingcon.com >>> >>> Web: http://www.wingcon.com >>> >>> >>> >>> Sitz der Gesellschaft: Langenargen >>> >>> Registergericht: ULM, HRB 632019 >>> >>> USt-Id.: DE232931635, WEEE-Id.: DE74015979 >>> >>> Geschäftsführer: Norbert Schäfer, Fritz R. Paul >>> >>> >>> >>> _______________________________________________ >>> >>> ceph-users mailing list >>> >>> ceph-users@lists.ceph.com >>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> >>> >>> >>> >>> -- >>> Mykola* * >> >> -- >> Mit freundlichen Grüßen >> >> David Riedl >> >> >> >> WINGcon GmbH Wireless New Generation - Consulting & Solutions >> >> Phone: +49 (0) 7543 9661 - 26 >> <tel:%2B49%20%280%29%207543%209661%20-%2026> >> E-Mail: david.ri...@wingcon.com <mailto:david.ri...@wingcon.com> >> Web: http://www.wingcon.com >> >> Sitz der Gesellschaft: Langenargen >> Registergericht: ULM, HRB 632019 >> USt-Id.: DE232931635, WEEE-Id.: DE74015979 >> Geschäftsführer: Norbert Schäfer, Fritz R. Paul >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> -- >> Mykola* * > > -- > Mit freundlichen Grüßen > > David Riedl > > > > WINGcon GmbH Wireless New Generation - Consulting & Solutions > > Phone: +49 (0) 7543 9661 - 26 > E-Mail: david.ri...@wingcon.com > Web: http://www.wingcon.com > > Sitz der Gesellschaft: Langenargen > Registergericht: ULM, HRB 632019 > USt-Id.: DE232931635, WEEE-Id.: DE74015979 > Geschäftsführer: Norbert Schäfer, Fritz R. Paul > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Steve Anthony LTS HPC Support Specialist Lehigh University sma...@lehigh.edu
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
