Rooney, Just tried hooking up osd.0 back. osd.0 seems to be better as I was able to run ceph-objectstore-tool export so decided to try hooking it up. Looks like journal is not happy. Is there any way to get this running? Or do I need to start getting data using ceph-objectstore-tool? 2017-09-13 18:51:50.051421 7f44dd847800 0 set uid:gid to 1001:1001 (ceph:ceph)2017-09-13 18:51:50.051435 7f44dd847800 0 ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0), process ceph-osd, pid 38992017-09-13 18:51:50.052323 7f44dd847800 0 pidfile_write: ignore empty --pid-file2017-09-13 18:51:50.061586 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)2017-09-13 18:51:50.061823 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option2017-09-13 18:51:50.061826 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option2017-09-13 18:51:50.061838 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice is supported2017-09-13 18:51:50.077506 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)2017-09-13 18:51:50.077549 7f44dd847800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf2017-09-13 18:51:50.078066 7f44dd847800 1 leveldb: Recovering log #280692017-09-13 18:51:50.177610 7f44dd847800 1 leveldb: Delete type=0 #28069 2017-09-13 18:51:50.177708 7f44dd847800 1 leveldb: Delete type=3 #28068 2017-09-13 18:51:57.946233 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled2017-09-13 18:51:57.947293 7f44dd847800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 18: 9998729216 bytes, block size 4096 bytes, directio = 1, aio = 12017-09-13 18:51:57.949835 7f44dd847800 -1 journal Unable to read past sequence 27057121 but header indicates the journal has committed up through 27057593, journal is corrupt2017-09-13 18:51:57.951824 7f44dd847800 -1 os/filestore/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f44dd847800 time 2017-09-13 18:51:57.949837os/filestore/FileJournal.cc: 2036: FAILED assert(0) ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x82) [0x55c809640d02] 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4] 3: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95] 4: (FileStore::mount()+0x2e28) [0x55c8092d0a88] 5: (OSD::init()+0x27d) [0x55c808f697ed] 6: (main()+0x2a64) [0x55c808ed05d4] 7: (__libc_start_main()+0xf5) [0x7f44da6e3b45] 8: (()+0x341117) [0x55c808f1b117] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -59> 2017-09-13 18:51:50.043283 7f44dd847800 5 asok(0x55c813d76000) register_command perfcounters_dump hook 0x55c813cbe030 -58> 2017-09-13 18:51:50.043312 7f44dd847800 5 asok(0x55c813d76000) register_command 1 hook 0x55c813cbe030 -57> 2017-09-13 18:51:50.043317 7f44dd847800 5 asok(0x55c813d76000) register_command perf dump hook 0x55c813cbe030 -56> 2017-09-13 18:51:50.043322 7f44dd847800 5 asok(0x55c813d76000) register_command perfcounters_schema hook 0x55c813cbe030 -55> 2017-09-13 18:51:50.043326 7f44dd847800 5 asok(0x55c813d76000) register_command 2 hook 0x55c813cbe030 -54> 2017-09-13 18:51:50.043330 7f44dd847800 5 asok(0x55c813d76000) register_command perf schema hook 0x55c813cbe030 -53> 2017-09-13 18:51:50.043334 7f44dd847800 5 asok(0x55c813d76000) register_command perf reset hook 0x55c813cbe030 -52> 2017-09-13 18:51:50.043339 7f44dd847800 5 asok(0x55c813d76000) register_command config show hook 0x55c813cbe030 -51> 2017-09-13 18:51:50.043344 7f44dd847800 5 asok(0x55c813d76000) register_command config set hook 0x55c813cbe030 -50> 2017-09-13 18:51:50.043349 7f44dd847800 5 asok(0x55c813d76000) register_command config get hook 0x55c813cbe030 -49> 2017-09-13 18:51:50.043355 7f44dd847800 5 asok(0x55c813d76000) register_command config diff hook 0x55c813cbe030 -48> 2017-09-13 18:51:50.043361 7f44dd847800 5 asok(0x55c813d76000) register_command log flush hook 0x55c813cbe030 -47> 2017-09-13 18:51:50.043367 7f44dd847800 5 asok(0x55c813d76000) register_command log dump hook 0x55c813cbe030 -46> 2017-09-13 18:51:50.043373 7f44dd847800 5 asok(0x55c813d76000) register_command log reopen hook 0x55c813cbe030 -45> 2017-09-13 18:51:50.051421 7f44dd847800 0 set uid:gid to 1001:1001 (ceph:ceph) -44> 2017-09-13 18:51:50.051435 7f44dd847800 0 ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0), process ceph-osd, pid 3899 -43> 2017-09-13 18:51:50.051455 7f44dd847800 5 object store type is filestore -42> 2017-09-13 18:51:50.052283 7f44dd847800 1 -- 192.168.1.30:0/0 learned my addr 192.168.1.30:0/0 -41> 2017-09-13 18:51:50.052290 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.1.30:6800/3899 need_addr=0 -40> 2017-09-13 18:51:50.052297 7f44dd847800 1 -- 192.168.2.30:0/0 learned my addr 192.168.2.30:0/0 -39> 2017-09-13 18:51:50.052300 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.2.30:6800/3899 need_addr=0 -38> 2017-09-13 18:51:50.052309 7f44dd847800 1 -- 192.168.2.30:0/0 learned my addr 192.168.2.30:0/0 -37> 2017-09-13 18:51:50.052313 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.2.30:6801/3899 need_addr=0 -36> 2017-09-13 18:51:50.052318 7f44dd847800 1 -- 192.168.1.30:0/0 learned my addr 192.168.1.30:0/0 -35> 2017-09-13 18:51:50.052321 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.1.30:6801/3899 need_addr=0 -34> 2017-09-13 18:51:50.052323 7f44dd847800 0 pidfile_write: ignore empty --pid-file -33> 2017-09-13 18:51:50.054120 7f44dd847800 5 asok(0x55c813d76000) init /var/run/ceph/ceph-osd.0.asok -32> 2017-09-13 18:51:50.054131 7f44dd847800 5 asok(0x55c813d76000) bind_and_listen /var/run/ceph/ceph-osd.0.asok -31> 2017-09-13 18:51:50.054205 7f44dd847800 5 asok(0x55c813d76000) register_command 0 hook 0x55c813cba0d0 -30> 2017-09-13 18:51:50.054211 7f44dd847800 5 asok(0x55c813d76000) register_command version hook 0x55c813cba0d0 -29> 2017-09-13 18:51:50.054215 7f44dd847800 5 asok(0x55c813d76000) register_command git_version hook 0x55c813cba0d0 -28> 2017-09-13 18:51:50.054219 7f44dd847800 5 asok(0x55c813d76000) register_command help hook 0x55c813cbe1e0 -27> 2017-09-13 18:51:50.054223 7f44dd847800 5 asok(0x55c813d76000) register_command get_command_descriptions hook 0x55c813cbe1f0 -26> 2017-09-13 18:51:50.054257 7f44d6fff700 5 asok(0x55c813d76000) entry start -25> 2017-09-13 18:51:50.054288 7f44dd847800 10 monclient(hunting): build_initial_monmap -24> 2017-09-13 18:51:50.061212 7f44dd847800 5 adding auth protocol: none -23> 2017-09-13 18:51:50.061222 7f44dd847800 5 adding auth protocol: none -22> 2017-09-13 18:51:50.061376 7f44dd847800 5 asok(0x55c813d76000) register_command objecter_requests hook 0x55c813cbe2b0 -21> 2017-09-13 18:51:50.061427 7f44dd847800 1 -- 192.168.1.30:6800/3899 messenger.start -20> 2017-09-13 18:51:50.061446 7f44dd847800 1 -- :/0 messenger.start -19> 2017-09-13 18:51:50.061460 7f44dd847800 1 -- 192.168.1.30:6801/3899 messenger.start -18> 2017-09-13 18:51:50.061472 7f44dd847800 1 -- 192.168.2.30:6801/3899 messenger.start -17> 2017-09-13 18:51:50.061484 7f44dd847800 1 -- 192.168.2.30:6800/3899 messenger.start -16> 2017-09-13 18:51:50.061497 7f44dd847800 1 -- :/0 messenger.start -15> 2017-09-13 18:51:50.061540 7f44dd847800 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal -14> 2017-09-13 18:51:50.061586 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) -13> 2017-09-13 18:51:50.061823 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -12> 2017-09-13 18:51:50.061826 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option -11> 2017-09-13 18:51:50.061838 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice is supported -10> 2017-09-13 18:51:50.077506 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -9> 2017-09-13 18:51:50.077549 7f44dd847800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf -8> 2017-09-13 18:51:50.078066 7f44dd847800 1 leveldb: Recovering log #28069 -7> 2017-09-13 18:51:50.177610 7f44dd847800 1 leveldb: Delete type=0 #28069 -6> 2017-09-13 18:51:50.177708 7f44dd847800 1 leveldb: Delete type=3 #28068 -5> 2017-09-13 18:51:57.946233 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled -4> 2017-09-13 18:51:57.947259 7f44dd847800 2 journal open /var/lib/ceph/osd/ceph-0/journal fsid f626317c-6e12-49d3-98ab-23d799430d30 fs_op_seq 27057120 -3> 2017-09-13 18:51:57.947293 7f44dd847800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 18: 9998729216 bytes, block size 4096 bytes, directio = 1, aio = 1 -2> 2017-09-13 18:51:57.949828 7f44dd847800 2 journal No further valid entries found, journal is most likely valid -1> 2017-09-13 18:51:57.949835 7f44dd847800 -1 journal Unable to read past sequence 27057121 but header indicates the journal has committed up through 27057593, journal is corrupt 0> 2017-09-13 18:51:57.951824 7f44dd847800 -1 os/filestore/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f44dd847800 time 2017-09-13 18:51:57.949837os/filestore/FileJournal.cc: 2036: FAILED assert(0) ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x82) [0x55c809640d02] 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4] 3: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95] 4: (FileStore::mount()+0x2e28) [0x55c8092d0a88] 5: (OSD::init()+0x27d) [0x55c808f697ed] 6: (main()+0x2a64) [0x55c808ed05d4] 7: (__libc_start_main()+0xf5) [0x7f44da6e3b45] 8: (()+0x341117) [0x55c808f1b117] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.--- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log--- end dump of recent events ---2017-09-13 18:51:57.954585 7f44dd847800 -1 *** Caught signal (Aborted) ** in thread 7f44dd847800 thread_name:ceph-osd ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0) 1: (()+0x960e57) [0x55c80953ae57] 2: (()+0xf8d0) [0x7f44dc6b58d0] 3: (gsignal()+0x37) [0x7f44da6f7067] 4: (abort()+0x148) [0x7f44da6f8448] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x55c809640ed6] 6: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4] 7: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95] 8: (FileStore::mount()+0x2e28) [0x55c8092d0a88] 9: (OSD::init()+0x27d) [0x55c808f697ed] 10: (main()+0x2a64) [0x55c808ed05d4] 11: (__libc_start_main()+0xf5) [0x7f44da6e3b45] 12: (()+0x341117) [0x55c808f1b117] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2017-09-13 18:51:57.954585 7f44dd847800 -1 *** Caught signal (Aborted) ** in thread 7f44dd847800 thread_name:ceph-osd ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0) 1: (()+0x960e57) [0x55c80953ae57] 2: (()+0xf8d0) [0x7f44dc6b58d0] 3: (gsignal()+0x37) [0x7f44da6f7067] 4: (abort()+0x148) [0x7f44da6f8448] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x55c809640ed6] 6: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4] 7: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95] 8: (FileStore::mount()+0x2e28) [0x55c8092d0a88] 9: (OSD::init()+0x27d) [0x55c808f697ed] 10: (main()+0x2a64) [0x55c808ed05d4] 11: (__libc_start_main()+0xf5) [0x7f44da6e3b45] 12: (()+0x341117) [0x55c808f1b117] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log--- end dump of recent events --- Regards,Hong
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
