Sorry for the broken post previously. I have looked into this more and
it looks like ceph-deploy is not seeing that it is a partition and
attempting to create an additional partition in the journals place. I
read in the documentation that if I set osd journal size = 0, that it
will assume that the target is a block device and use the entire block.
I tried this and it still doesn't work. I have since zapped the journals
and specified a 20G journal size. Now in my ceph-deploy line I just
specify :
ceph-deploy osd --dmcrypt --fs-type ${fs} create
${host}:${disk}:/dev/${journal_disk}
IE::
ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-1:sde:/dev/sdab
ceph-deploy osd --dmcrypt --fs-type btrfs create kh28-1:sdf:/dev/sdab
and ceph-deploy seems to try to create a new partition every time.
I have now run into a new issue though. After ceph-deploy creates the
partitions and seems to bootstrap the disks successfully it doest not
mount them properly to create the journal.
[ceph_deploy.osd][DEBUG ] Calling partprobe on zapped device /dev/sdr
[1565/1920]
[kh28-3.osdc.io][INFO ] Running command: sudo partprobe /dev/sdr
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/lacadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.25): /usr/local/bin/ceph-deploy
osd --dmcrypt --fs-type btrfs create kh28-3.osdc.io:sdr:/dev/sdp2
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
kh28-3.osdc.io:/dev/sdr:/dev/sdp2
[kh28-3.osdc.io][DEBUG ] connection detected need for sudo
[kh28-3.osdc.io][DEBUG ] connected to host: kh28-3.osdc.io
[kh28-3.osdc.io][DEBUG ] detect platform information from remote host
[kh28-3.osdc.io][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to kh28-3.osdc.io
[kh28-3.osdc.io][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf
[kh28-3.osdc.io][INFO ] Running command: sudo udevadm trigger
--subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host kh28-3.osdc.io disk /dev/sdr
journal /dev/sdp2 activate True
[kh28-3.osdc.io][INFO ] Running command: sudo ceph-disk -v prepare
--fs-type btrfs --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys
--cluster ceph -- /dev/sdr /dev/sdp2
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_mkfs_options_btrfs
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_fs_mkfs_options_btrfs
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_mount_options_btrfs
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_cryptsetup_parameters
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating journal file /dev/sdp2
with size 0 (ceph-osd will resize and allocate)
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Journal is file /dev/sdp2
[kh28-3.osdc.io][WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable
if journal is not the same device as the osd data
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk
--largest-new=1 --change-name=1:ceph data
--partition-guid=1:c1879421-bcd0-4419-bc96-63d2d51176db
--typecode=1:89c57f98-2fe5-4dc0-89c1-5ec00ceff2be -- /dev/sdr
[kh28-3.osdc.io][DEBUG ] The operation has completed successfully.
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Calling partprobe on created
device /dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe
/dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm
settle
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/sbin/cryptsetup --batch-mode --key-file
/etc/ceph/dmcrypt-keys/c1879421-bcd0-4419-bc96-63d2d51176db.luks.key
luksFormat /dev/sdr1
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/sbin/cryptsetup --key-file
/etc/ceph/dmcrypt-keys/c1879421-bcd0-4419-bc96-63d2d51176db.luks.key
luksOpen /dev/sdr1 c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating btrfs fs on
/dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t
btrfs -m single -l 32768 -n 32768 --
/dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] Turning ON incompat feature 'extref': increased
hardlink limit per file to 65536
[kh28-3.osdc.io][DEBUG ]
[kh28-3.osdc.io][DEBUG ] WARNING! - Btrfs v3.12 IS EXPERIMENTAL
[kh28-3.osdc.io][DEBUG ] WARNING! - see http://btrfs.wiki.kernel.org
before using
[kh28-3.osdc.io][DEBUG ]
[kh28-3.osdc.io][DEBUG ] fs created label (null) on
/dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][DEBUG ] nodesize 32768 leafsize 32768 sectorsize
4096 size 3.64TiB
[kh28-3.osdc.io][DEBUG ] Btrfs v3.12
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Mounting
/dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db on
/var/lib/ceph/tmp/mnt.lYDwEH with options
noatime,autodefrag,space_cache,recovery
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t
btrfs -o noatime,autodefrag,space_cache,recovery --
/dev/mapper/c1879421-bcd0-4419-bc96-63d2d51176db
/var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Preparing osd data dir
/var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Creating symlink
/var/lib/ceph/tmp/mnt.lYDwEH/journal -> /dev/sdp2
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Unmounting
/var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /bin/umount --
/var/lib/ceph/tmp/mnt.lYDwEH
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command:
/sbin/cryptsetup remove c1879421-bcd0-4419-bc96-63d2d51176db
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk
--typecode=1:4fbd7e29-9d25-41b8-afd0-35865ceff05d -- /dev/sdr
[kh28-3.osdc.io][DEBUG ] The operation has completed successfully.
[kh28-3.osdc.io][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared
device /dev/sdr
[kh28-3.osdc.io][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe
/dev/sdr
[kh28-3.osdc.io][INFO ] Running command: sudo udevadm trigger
--subsystem-match=block --action=add
[kh28-3.osdc.io][INFO ] checking OSD status...
[kh28-3.osdc.io][INFO ] Running command: sudo ceph --cluster=ceph osd
stat --format=json
[kh28-3.osdc.io][WARNIN] there are 36 OSDs down
[kh28-3.osdc.io][WARNIN] there are 36 OSDs out
[ceph_deploy.osd][DEBUG ] Host kh28-3.osdc.io is now ready for osd use.
I've tried this with xfs as well and it doesn't seem to make a
difference. Checking the log for any of the osds on this host in
/var/log/ceph/ seems to show that the journal is not created properly::
trying XFS::
2015-07-23 15:18:47.236679 7efd5aeab900 0 ceph version 0.94.2
(5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 973519
2015-07-23 15:18:47.239134 7efd5aeab900 1
filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkfs in /var/lib/ceph/tmp/mnt.TZy7HL
2015-07-23 15:18:47.239156 7efd5aeab900 1
filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkfs fsid is already set to
a473d735-672f-4835-9217-55b084ad5a35
2015-07-23 15:18:47.239554 7efd5aeab900 0
filestore(/var/lib/ceph/tmp/mnt.TZy7HL) backend xfs (magic 0x58465342)
2015-07-23 15:18:47.363235 7efd5aeab900 1
filestore(/var/lib/ceph/tmp/mnt.TZy7HL) leveldb db exists/created
2015-07-23 15:18:47.363330 7efd5aeab900 -1 journal FileJournal::_open:
disabling aio for non-block journal. Use journal_force_aio to force use
of aio anyway
2015-07-23 15:18:47.363337 7efd5aeab900 1 journal _open
/var/lib/ceph/tmp/mnt.TZy7HL/journal fd 10: 0 bytes, block size 4096
bytes, directio = 1, aio = 0
2015-07-23 15:18:47.363357 7efd5aeab900 -1 journal check: ondisk fsid
00000000-0000-0000-0000-000000000000 doesn't match expected
a473d735-672f-4835-9217-55b084ad5a35, invalid (someone else's?) journal
2015-07-23 15:18:47.363385 7efd5aeab900 -1
filestore(/var/lib/ceph/tmp/mnt.TZy7HL) mkjournal error creating journal
on /var/lib/ceph/tmp/mnt.TZy7HL/journal: (22) Invalid argument
2015-07-23 15:18:47.363406 7efd5aeab900 -1 OSD::mkfs: ObjectStore::mkfs
failed with error -22
2015-07-23 15:18:47.363439 7efd5aeab900 -1 ** ERROR: error creating
empty object store in /var/lib/ceph/tmp/mnt.TZy7HL: (22) Invalid argument
trying btrfs::
2015-07-23 16:40:12.429496 7fef6fbb2900 -1 ** ERROR: error creating
empty object store in /var/lib/ceph/tmp/mnt.cbhhm7: (22) Invalid argument
2015-07-23 16:40:31.380336 7fb07eff1900 0 ceph version 0.94.2
(5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1153140
2015-07-23 16:40:31.382097 7fb07eff1900 1
filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkfs in /var/lib/ceph/tmp/mnt.Aa9khA
2015-07-23 16:40:31.382118 7fb07eff1900 1
filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkfs fsid is already set to
f8b57033-dd8d-41ea-ace1-593bd6048789
2015-07-23 16:40:31.382193 7fb07eff1900 0
filestore(/var/lib/ceph/tmp/mnt.Aa9khA) backend btrfs (magic 0x9123683e)
2015-07-23 16:40:31.411131 7fb07eff1900 1
filestore(/var/lib/ceph/tmp/mnt.Aa9khA) leveldb db exists/created
2015-07-23 16:40:31.411203 7fb07eff1900 -1 journal FileJournal::_open:
disabling aio for non-block journal. Use journal_force_aio to force use
of aio anyway
2015-07-23 16:40:31.411211 7fb07eff1900 1 journal _open
/var/lib/ceph/tmp/mnt.Aa9khA/journal fd 10: 0 bytes, block size 4096
bytes, directio = 1, aio = 0
2015-07-23 16:40:31.411225 7fb07eff1900 -1 journal check: ondisk fsid
00000000-0000-0000-0000-000000000000 doesn't match expected
f8b57033-dd8d-41ea-ace1-593bd6048789, invalid (someone else's?) journal
2015-07-23 16:40:31.411251 7fb07eff1900 -1
filestore(/var/lib/ceph/tmp/mnt.Aa9khA) mkjournal error creating journal
on /var/lib/ceph/tmp/mnt.Aa9khA/journal: (22) Invalid argument
2015-07-23 16:40:31.411263 7fb07eff1900 -1 OSD::mkfs: ObjectStore::mkfs
failed with error -22
2015-07-23 16:40:31.411291 7fb07eff1900 -1 ** ERROR: error creating
empty object store in /var/lib/ceph/tmp/mnt.Aa9khA: (22) Invalid argument
What is strange is that this all seemed to work without issue in Giant
so I am not sure what has changed. I am about to try a manual deploy
next but i've grown lazy with ceph-deploy.
Has anyone experienced anything similar or know a work around (outside
of bootstrapping and deploying each osd manually?)
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
