[ceph-users] Fresh deploy of ceph 0.83 has OSD down

Mark Kirkwood Wed, 06 Aug 2014 16:07:20 -0700

Hi,

I'm doing a fresh install of ceph 0.83 (src build) to an Ubuntu 14.04 VMusing ceph-deploy 1.59. Everything goes well until the osd creation,which fails to start with a journal open error. The steps are shownbelow (ceph is the deploy target host):



(ceph1) $ uname -a

Linux ceph1 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC2014 x86_64 x86_64 x86_64 GNU/Linux

(ceph1) $ ceph -v
ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81)

$ ceph-deploy --version
1.5.9
$ ceph-deploy -v new ceph1
$ vi ceph.conf
$ cat ceph.conf
[global]
fsid = 624d6d49-c090-4bfe-a71d-a54b7e13c037
mon_initial_members = ceph1
mon_host = 192.168.122.21
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_journal_size = 2048

$ ceph-deploy -v mon create ceph1
$ ceph-deploy -v gatherkeys ceph1

$ ceph-deploy -v disk zap ceph1:/dev/vdb
$ ceph-deploy -v disk zap ceph1:/dev/vdc

$ ceph-deploy -v osd create ceph1:/dev/vdb:/dev/vdc
...
[ceph1][WARNIN] there is 1 OSD down
[ceph1][WARNIN] there is 1 OSD out

$ tail ceph.osd.0.log

2014-08-07 10:47:45.350623 7ffe95e05800 1 journal _open/var/lib/ceph/osd/ceph-0/journal fd 20: 2147483648 bytes, block size4096 bytes, directio = 1, aio = 12014-08-07 10:47:45.351364 7ffe95e05800 -1 journal read_header errordecoding journal header2014-08-07 10:47:45.351398 7ffe95e05800 -1filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal/var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument

2014-08-07 10:47:38.506876 7f36e482d800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 1765
2014-08-07 10:47:38.521928 7f36e482d800  1 journal _open /dev/vdc1 fd 4: 2147483648 bytes, block size 4096 bytes, directio = 0, aio = 0
2014-08-07 10:47:41.934775 7f9544ffe800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 1885
2014-08-07 10:47:41.939595 7f9544ffe800  1 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) mkfs in /var/lib/ceph/tmp/mnt.UXqBUb
2014-08-07 10:47:41.939647 7f9544ffe800  1 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) mkfs fsid is already set to 60c287fc-3544-4830-bec8-2d2dc4e449e5
2014-08-07 10:47:42.005465 7f9544ffe800  0 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) backend xfs (magic 0x58465342)
2014-08-07 10:47:42.005476 7f9544ffe800  1 filestore(/var/lib/ceph/tmp/mnt.UXqBUb)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-08-07 10:47:42.065586 7f9544ffe800  1 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) leveldb db exists/created
2014-08-07 10:47:42.067012 7f9544ffe800  1 journal _open /var/lib/ceph/tmp/mnt.UXqBUb/journal fd 10: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:42.067312 7f9544ffe800 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 60c287fc-3544-4830-bec8-2d2dc4e449e5, invalid (someone else's?) journal
2014-08-07 10:47:42.068530 7f9544ffe800  1 journal _open /var/lib/ceph/tmp/mnt.UXqBUb/journal fd 10: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:42.072536 7f9544ffe800  0 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) mkjournal created journal on /var/lib/ceph/tmp/mnt.UXqBUb/journal
2014-08-07 10:47:42.072574 7f9544ffe800  1 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) mkfs done in /var/lib/ceph/tmp/mnt.UXqBUb
2014-08-07 10:47:42.072688 7f9544ffe800  0 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) backend xfs (magic 0x58465342)
2014-08-07 10:47:42.095571 7f9544ffe800  0 genericfilestorebackend(/var/lib/ceph/tmp/mnt.UXqBUb) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-07 10:47:42.095595 7f9544ffe800  0 genericfilestorebackend(/var/lib/ceph/tmp/mnt.UXqBUb) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-07 10:47:42.107497 7f9544ffe800  0 genericfilestorebackend(/var/lib/ceph/tmp/mnt.UXqBUb) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-07 10:47:42.107571 7f9544ffe800  0 xfsfilestorebackend(/var/lib/ceph/tmp/mnt.UXqBUb) detect_feature: extsize is disabled by conf
2014-08-07 10:47:42.144479 7f9544ffe800  0 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-07 10:47:42.147178 7f9544ffe800  1 journal _open /var/lib/ceph/tmp/mnt.UXqBUb/journal fd 16: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:42.150303 7f9544ffe800  1 journal _open /var/lib/ceph/tmp/mnt.UXqBUb/journal fd 16: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:42.150665 7f9544ffe800 -1 filestore(/var/lib/ceph/tmp/mnt.UXqBUb) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-08-07 10:47:42.276297 7f9544ffe800  1 journal close /var/lib/ceph/tmp/mnt.UXqBUb/journal
2014-08-07 10:47:42.276790 7f9544ffe800 -1 created object store /var/lib/ceph/tmp/mnt.UXqBUb journal /var/lib/ceph/tmp/mnt.UXqBUb/journal for osd.0 fsid 624d6d49-c090-4bfe-a71d-a54b7e13c037
2014-08-07 10:47:42.276845 7f9544ffe800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.UXqBUb/keyring: can't open /var/lib/ceph/tmp/mnt.UXqBUb/keyring: (2) No such file or directory
2014-08-07 10:47:42.277284 7f9544ffe800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.UXqBUb/keyring
2014-08-07 10:47:42.852010 7fbc256ea800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 2057
2014-08-07 10:47:42.854667 7fbc256ea800  1 journal _open /dev/vdc1 fd 4: 2147483648 bytes, block size 4096 bytes, directio = 0, aio = 0
2014-08-07 10:47:42.862774 7fbc256ea800 -1 journal read_header error decoding journal header
2014-08-07 10:47:43.892490 7f6ff6502800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 2063
2014-08-07 10:47:43.893650 7f6ff6502800  0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2014-08-07 10:47:43.893655 7f6ff6502800  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-08-07 10:47:43.920541 7f6ff6502800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-07 10:47:43.920549 7f6ff6502800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-07 10:47:43.947080 7f6ff6502800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-07 10:47:43.947153 7f6ff6502800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
2014-08-07 10:47:44.016615 7f6ff6502800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-07 10:47:44.019548 7f6ff6502800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:44.021130 7f6ff6502800 -1 journal read_header error decoding journal header
2014-08-07 10:47:44.021200 7f6ff6502800 -1 filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2014-08-07 10:47:44.028493 7f6ff6502800 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-0: (22) Invalid argument[0m
2014-08-07 10:47:44.200093 7f00c1129800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 2128
2014-08-07 10:47:44.200704 7f00c1129800  0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2014-08-07 10:47:44.200709 7f00c1129800  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-08-07 10:47:44.226775 7f00c1129800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-07 10:47:44.226782 7f00c1129800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-07 10:47:44.244274 7f00c1129800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-07 10:47:44.244318 7f00c1129800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
2014-08-07 10:47:44.292723 7f00c1129800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-07 10:47:44.295585 7f00c1129800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:44.296542 7f00c1129800 -1 journal read_header error decoding journal header
2014-08-07 10:47:44.296605 7f00c1129800 -1 filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2014-08-07 10:47:44.304559 7f00c1129800 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-0: (22) Invalid argument[0m
2014-08-07 10:47:44.467515 7fad529a9800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 2193
2014-08-07 10:47:44.467994 7fad529a9800  0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2014-08-07 10:47:44.467999 7fad529a9800  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-08-07 10:47:44.496844 7fad529a9800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-07 10:47:44.496860 7fad529a9800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-07 10:47:44.514279 7fad529a9800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-07 10:47:44.514350 7fad529a9800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
2014-08-07 10:47:44.562865 7fad529a9800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-07 10:47:44.566564 7fad529a9800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:44.567437 7fad529a9800 -1 journal read_header error decoding journal header
2014-08-07 10:47:44.567519 7fad529a9800 -1 filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2014-08-07 10:47:44.574676 7fad529a9800 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-0: (22) Invalid argument[0m
2014-08-07 10:47:44.738357 7f6b32fba800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 2258
2014-08-07 10:47:44.738832 7f6b32fba800  0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2014-08-07 10:47:44.738837 7f6b32fba800  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-08-07 10:47:44.766479 7f6b32fba800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-07 10:47:44.766491 7f6b32fba800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-07 10:47:44.784235 7f6b32fba800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-07 10:47:44.784282 7f6b32fba800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
2014-08-07 10:47:44.832546 7f6b32fba800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-07 10:47:44.834281 7f6b32fba800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:44.835003 7f6b32fba800 -1 journal read_header error decoding journal header
2014-08-07 10:47:44.835034 7f6b32fba800 -1 filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2014-08-07 10:47:44.843733 7f6b32fba800 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-0: (22) Invalid argument[0m
2014-08-07 10:47:44.991554 7f3f3c7a1800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 2323
2014-08-07 10:47:44.992040 7f3f3c7a1800  0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2014-08-07 10:47:44.992045 7f3f3c7a1800  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-08-07 10:47:45.018569 7f3f3c7a1800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-07 10:47:45.018586 7f3f3c7a1800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-07 10:47:45.048199 7f3f3c7a1800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-07 10:47:45.048247 7f3f3c7a1800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
2014-08-07 10:47:45.096576 7f3f3c7a1800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-07 10:47:45.098666 7f3f3c7a1800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:45.099404 7f3f3c7a1800 -1 journal read_header error decoding journal header
2014-08-07 10:47:45.099440 7f3f3c7a1800 -1 filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2014-08-07 10:47:45.108088 7f3f3c7a1800 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-0: (22) Invalid argument[0m
2014-08-07 10:47:45.255013 7ffe95e05800  0 ceph version 0.83-399-gf77449c (f77449cb4bc6dff36264af6983d345bab3b95c81), process ceph-osd, pid 2388
2014-08-07 10:47:45.255497 7ffe95e05800  0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2014-08-07 10:47:45.255502 7ffe95e05800  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-08-07 10:47:45.282586 7ffe95e05800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-07 10:47:45.282596 7ffe95e05800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-07 10:47:45.300394 7ffe95e05800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-07 10:47:45.300435 7ffe95e05800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
2014-08-07 10:47:45.348642 7ffe95e05800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-07 10:47:45.350623 7ffe95e05800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 20: 2147483648 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-07 10:47:45.351364 7ffe95e05800 -1 journal read_header error decoding journal header
2014-08-07 10:47:45.351398 7ffe95e05800 -1 filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2014-08-07 10:47:45.360158 7ffe95e05800 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-0: (22) Invalid argument[0m

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Fresh deploy of ceph 0.83 has OSD down

Reply via email to