I've finally learned enough about the OSD backend track down this issue to what I believe is the root cause. LevelDB compaction is the common thread every time we move data around our cluster. I've ruled out PG subfolder splitting, EC doesn't seem to be the root cause of this, and it is cluster wide as opposed to specific hardware.
One of the first things I found after digging into leveldb omap compaction was [1] this article with a heading "RocksDB instead of LevelDB" which mentions that leveldb was replaced with rocksdb as the default db backend for filestore OSDs and was even backported to Jewel because of the performance improvements. I figured there must be a way to be able to upgrade an OSD to use rocksdb from leveldb without needing to fully backfill the entire OSD. There is [2] this article, but you need to have an active service account with RedHat to access it. I eventually came across [3] this article about optimizing Ceph Object Storage which mentions a resolution to OSDs flapping due to omap compaction to migrate to using rocksdb. It links to the RedHat article, but also has [4] these steps outlined in it. I tried to follow the steps, but the OSD I tested this on was unable to start with [5] this segfault. And then trying to move the OSD back to the original LevelDB omap folder resulted in [6] this in the log. I apologize that all of my logging is with log level 1. If needed I can get some higher log levels. My Ceph version is 12.2.4. Does anyone have any suggestions for how I can update my filestore backend from leveldb to rocksdb? Or if that's the wrong direction and I should be looking elsewhere? Thank you. [1] https://ceph.com/community/new-luminous-rados-improvements/ [2] https://access.redhat.com/solutions/3210951 [3] https://hubb.blob.core.windows.net/c2511cea-81c5-4386-8731-cc444ff806df-public/resources/Optimize%20Ceph%20object%20storage%20for%20production%20in%20multisite%20clouds.pdf [4] ■ Stop the OSD ■ mv /var/lib/ceph/osd/ceph-/current/omap /var/lib/ceph/osd/ceph-/omap.orig ■ ulimit -n 65535 ■ ceph-kvstore-tool leveldb /var/lib/ceph/osd/ceph-/omap.orig store-copy /var/lib/ceph/osd/ceph-/current/omap 10000 rocksdb ■ ceph-osdomap-tool --omap-path /var/lib/ceph/osd/ceph-/current/omap --command check ■ sed -i s/leveldb/rocksdb/g /var/lib/ceph/osd/ceph-/superblock ■ chown ceph.ceph /var/lib/ceph/osd/ceph-/current/omap -R ■ cd /var/lib/ceph/osd/ceph-; rm -rf omap.orig ■ Start the OSD [5] 2018-09-17 19:23:10.826227 7f1f3f2ab700 -1 abort: Corruption: Snappy not supported or corrupted Snappy compressed block contents 2018-09-17 19:23:10.830525 7f1f3f2ab700 -1 *** Caught signal (Aborted) ** [6] 2018-09-17 19:27:34.010125 7fcdee97cd80 -1 osd.0 0 OSD:init: unable to mount object store 2018-09-17 19:27:34.010131 7fcdee97cd80 -1 ESC[0;31m ** ERROR: osd init failed: (1) Operation not permittedESC[0m 2018-09-17 19:27:54.225941 7f7f03308d80 0 set uid:gid to 167:167 (ceph:ceph) 2018-09-17 19:27:54.225975 7f7f03308d80 0 ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable), process (unknown), pid 361535 2018-09-17 19:27:54.231275 7f7f03308d80 0 pidfile_write: ignore empty --pid-file 2018-09-17 19:27:54.260207 7f7f03308d80 0 load: jerasure load: lrc load: isa 2018-09-17 19:27:54.260520 7f7f03308d80 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2018-09-17 19:27:54.261135 7f7f03308d80 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2018-09-17 19:27:54.261750 7f7f03308d80 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2018-09-17 19:27:54.261757 7f7f03308d80 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2018-09-17 19:27:54.261758 7f7f03308d80 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice() is disabled via 'filestore splice' config option 2018-09-17 19:27:54.286454 7f7f03308d80 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2018-09-17 19:27:54.286572 7f7f03308d80 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf 2018-09-17 19:27:54.287119 7f7f03308d80 0 filestore(/var/lib/ceph/osd/ceph-0) start omap initiation 2018-09-17 19:27:54.287527 7f7f03308d80 -1 filestore(/var/lib/ceph/osd/ceph-0) mount(1723): Error initializing leveldb : Corruption: VersionEdit: unknown tag
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
