Repair has gone farther but failed on something different - this time it appears to be related to store inconsistency rather than lack of free space. Emailed log to you, beware: over 2GB uncompressed.
> On 3.10.2018, at 13:15, Igor Fedotov <ifedo...@suse.de> wrote: > > You may want to try new updates from the PR along with disabling flush on > recovery for rocksdb (avoid_flush_during_recovery parameter). > > Full cmd line might looks like: > > CEPH_ARGS="--bluestore_rocksdb_options avoid_flush_during_recovery=1" > bin/ceph-bluestore-tool --path <osd path> repair > > > To be applied for "non-expanded" OSDs where repair didn't pass. > > Please collect a log during repair... > > > Thanks, > > Igor > > On 10/2/2018 4:32 PM, Sergey Malinin wrote: >> Repair goes through only when LVM volume has been expanded, otherwise it >> fails with enospc as well as any other operation. However, expanding the >> volume immediately renders bluefs unmountable with IO error. >> 2 of 3 OSDs got bluefs log currupted (bluestore tool segfaults at the very >> end of bluefs-log-dump), I'm not sure whether corruption occurred before or >> after volume expansion. >> >> >>> On 2.10.2018, at 16:07, Igor Fedotov <ifedo...@suse.de> wrote: >>> >>> You mentioned repair had worked before, is that correct? What's the >>> difference now except the applied patch? Different OSD? Anything else? >>> >>> >>> On 10/2/2018 3:52 PM, Sergey Malinin wrote: >>> >>>> It didn't work, emailed logs to you. >>>> >>>> >>>>> On 2.10.2018, at 14:43, Igor Fedotov <ifedo...@suse.de> wrote: >>>>> >>>>> The major change is in get_bluefs_rebalance_txn function, it lacked >>>>> bluefs_rebalance_txn assignment.. >>>>> >>>>> >>>>> >>>>> On 10/2/2018 2:40 PM, Sergey Malinin wrote: >>>>>> PR doesn't seem to have changed since yesterday. Am I missing something? >>>>>> >>>>>> >>>>>>> On 2.10.2018, at 14:15, Igor Fedotov <ifedo...@suse.de> wrote: >>>>>>> >>>>>>> Please update the patch from the PR - it didn't update bluefs extents >>>>>>> list before. >>>>>>> >>>>>>> Also please set debug bluestore 20 when re-running repair and collect >>>>>>> the log. >>>>>>> >>>>>>> If repair doesn't help - would you send repair and startup logs >>>>>>> directly to me as I have some issues accessing ceph-post-file uploads. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Igor >>>>>>> >>>>>>> >>>>>>> On 10/2/2018 11:39 AM, Sergey Malinin wrote: >>>>>>>> Yes, I did repair all OSDs and it finished with 'repair success'. I >>>>>>>> backed up OSDs so now I have more room to play. >>>>>>>> I posted log files using ceph-post-file with the following IDs: >>>>>>>> 4af9cc4d-9c73-41c9-9c38-eb6c551047a0 >>>>>>>> 20df7df5-f0c9-4186-aa21-4e5c0172cd93 >>>>>>>> >>>>>>>> >>>>>>>>> On 2.10.2018, at 11:26, Igor Fedotov <ifedo...@suse.de> wrote: >>>>>>>>> >>>>>>>>> You did repair for any of this OSDs, didn't you? For all of them? >>>>>>>>> >>>>>>>>> >>>>>>>>> Would you please provide the log for both types (failed on mount and >>>>>>>>> failed with enospc) of failing OSDs. Prior to collecting please >>>>>>>>> remove existing ones prior and set debug bluestore to 20. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10/2/2018 2:16 AM, Sergey Malinin wrote: >>>>>>>>>> I was able to apply patches to mimic, but nothing changed. One osd >>>>>>>>>> that I had space expanded on fails with bluefs mount IO error, >>>>>>>>>> others keep failing with enospc. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 1.10.2018, at 19:26, Igor Fedotov <ifedo...@suse.de> wrote: >>>>>>>>>>> >>>>>>>>>>> So you should call repair which rebalances (i.e. allocates >>>>>>>>>>> additional space) BlueFS space. Hence allowing OSD to start. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Igor >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 10/1/2018 7:22 PM, Igor Fedotov wrote: >>>>>>>>>>>> Not exactly. The rebalancing from this kv_sync_thread still might >>>>>>>>>>>> be deferred due to the nature of this thread (haven't 100% sure >>>>>>>>>>>> though). >>>>>>>>>>>> >>>>>>>>>>>> Here is my PR showing the idea (still untested and perhaps >>>>>>>>>>>> unfinished!!!) >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/ceph/ceph/pull/24353 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Igor >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 10/1/2018 7:07 PM, Sergey Malinin wrote: >>>>>>>>>>>>> Can you please confirm whether I got this right: >>>>>>>>>>>>> >>>>>>>>>>>>> --- BlueStore.cc.bak 2018-10-01 18:54:45.096836419 +0300 >>>>>>>>>>>>> +++ BlueStore.cc 2018-10-01 19:01:35.937623861 +0300 >>>>>>>>>>>>> @@ -9049,22 +9049,17 @@ >>>>>>>>>>>>> throttle_bytes.put(costs); >>>>>>>>>>>>> PExtentVector bluefs_gift_extents; >>>>>>>>>>>>> - if (bluefs && >>>>>>>>>>>>> - after_flush - bluefs_last_balance > >>>>>>>>>>>>> - cct->_conf->bluestore_bluefs_balance_interval) { >>>>>>>>>>>>> - bluefs_last_balance = after_flush; >>>>>>>>>>>>> - int r = _balance_bluefs_freespace(&bluefs_gift_extents); >>>>>>>>>>>>> - assert(r >= 0); >>>>>>>>>>>>> - if (r > 0) { >>>>>>>>>>>>> - for (auto& p : bluefs_gift_extents) { >>>>>>>>>>>>> - bluefs_extents.insert(p.offset, p.length); >>>>>>>>>>>>> - } >>>>>>>>>>>>> - bufferlist bl; >>>>>>>>>>>>> - encode(bluefs_extents, bl); >>>>>>>>>>>>> - dout(10) << __func__ << " bluefs_extents now 0x" << >>>>>>>>>>>>> std::hex >>>>>>>>>>>>> - << bluefs_extents << std::dec << dendl; >>>>>>>>>>>>> - synct->set(PREFIX_SUPER, "bluefs_extents", bl); >>>>>>>>>>>>> + int r = _balance_bluefs_freespace(&bluefs_gift_extents); >>>>>>>>>>>>> + ceph_assert(r >= 0); >>>>>>>>>>>>> + if (r > 0) { >>>>>>>>>>>>> + for (auto& p : bluefs_gift_extents) { >>>>>>>>>>>>> + bluefs_extents.insert(p.offset, p.length); >>>>>>>>>>>>> } >>>>>>>>>>>>> + bufferlist bl; >>>>>>>>>>>>> + encode(bluefs_extents, bl); >>>>>>>>>>>>> + dout(10) << __func__ << " bluefs_extents now 0x" << std::hex >>>>>>>>>>>>> + << bluefs_extents << std::dec << dendl; >>>>>>>>>>>>> + synct->set(PREFIX_SUPER, "bluefs_extents", bl); >>>>>>>>>>>>> } >>>>>>>>>>>>> // cleanup sync deferred keys >>>>>>>>>>>>> >>>>>>>>>>>>>> On 1.10.2018, at 18:39, Igor Fedotov <ifedo...@suse.de> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> So you have just a single main device per OSD.... >>>>>>>>>>>>>> >>>>>>>>>>>>>> Then bluestore-tool wouldn't help, it's unable to expand BlueFS >>>>>>>>>>>>>> partition at main device, standalone devices are supported only. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Given that you're able to rebuild the code I can suggest to make >>>>>>>>>>>>>> a patch that triggers BlueFS rebalance (see code snippet below) >>>>>>>>>>>>>> on repairing. >>>>>>>>>>>>>> PExtentVector bluefs_gift_extents; >>>>>>>>>>>>>> int r = _balance_bluefs_freespace(&bluefs_gift_extents); >>>>>>>>>>>>>> ceph_assert(r >= 0); >>>>>>>>>>>>>> if (r > 0) { >>>>>>>>>>>>>> for (auto& p : bluefs_gift_extents) { >>>>>>>>>>>>>> bluefs_extents.insert(p.offset, p.length); >>>>>>>>>>>>>> } >>>>>>>>>>>>>> bufferlist bl; >>>>>>>>>>>>>> encode(bluefs_extents, bl); >>>>>>>>>>>>>> dout(10) << __func__ << " bluefs_extents now 0x" << >>>>>>>>>>>>>> std::hex >>>>>>>>>>>>>> << bluefs_extents << std::dec << dendl; >>>>>>>>>>>>>> synct->set(PREFIX_SUPER, "bluefs_extents", bl); >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> If it waits I can probably make a corresponding PR tomorrow. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Igor >>>>>>>>>>>>>> On 10/1/2018 6:16 PM, Sergey Malinin wrote: >>>>>>>>>>>>>>> I have rebuilt the tool, but none of my OSDs no matter dead or >>>>>>>>>>>>>>> alive have any symlinks other than 'block' pointing to LVM. >>>>>>>>>>>>>>> I adjusted main device size but it looks like it needs even >>>>>>>>>>>>>>> more space for db compaction. After executing >>>>>>>>>>>>>>> bluefs-bdev-expand OSD fails to start, however 'fsck' and >>>>>>>>>>>>>>> 'repair' commands finished successfully. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2018-10-01 18:02:39.755 7fc9226c6240 1 freelist init >>>>>>>>>>>>>>> 2018-10-01 18:02:39.763 7fc9226c6240 1 >>>>>>>>>>>>>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc opening >>>>>>>>>>>>>>> allocation metadata >>>>>>>>>>>>>>> 2018-10-01 18:02:40.907 7fc9226c6240 1 >>>>>>>>>>>>>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 285 GiB >>>>>>>>>>>>>>> in 2249899 extents >>>>>>>>>>>>>>> 2018-10-01 18:02:40.951 7fc9226c6240 -1 >>>>>>>>>>>>>>> bluestore(/var/lib/ceph/osd/ceph-1) _reconcile_bluefs_freespace >>>>>>>>>>>>>>> bluefs extra 0x[6d6f000000~50c800000] >>>>>>>>>>>>>>> 2018-10-01 18:02:40.951 7fc9226c6240 1 stupidalloc >>>>>>>>>>>>>>> 0x0x55d053fb9180 shutdown >>>>>>>>>>>>>>> 2018-10-01 18:02:40.963 7fc9226c6240 1 freelist shutdown >>>>>>>>>>>>>>> 2018-10-01 18:02:40.963 7fc9226c6240 4 rocksdb: >>>>>>>>>>>>>>> [/build/ceph-13.2.2/src/rocksdb/db/db_impl.cc:252] Shutdown: >>>>>>>>>>>>>>> canceling all background work >>>>>>>>>>>>>>> 2018-10-01 18:02:40.967 7fc9226c6240 4 rocksdb: >>>>>>>>>>>>>>> [/build/ceph-13.2.2/src/rocksdb/db/db_impl.cc:397] Shutdown >>>>>>>>>>>>>>> complete >>>>>>>>>>>>>>> 2018-10-01 18:02:40.971 7fc9226c6240 1 bluefs umount >>>>>>>>>>>>>>> 2018-10-01 18:02:40.975 7fc9226c6240 1 stupidalloc >>>>>>>>>>>>>>> 0x0x55d053883800 shutdown >>>>>>>>>>>>>>> 2018-10-01 18:02:40.975 7fc9226c6240 1 bdev(0x55d053c32e00 >>>>>>>>>>>>>>> /var/lib/ceph/osd/ceph-1/block) close >>>>>>>>>>>>>>> 2018-10-01 18:02:41.267 7fc9226c6240 1 bdev(0x55d053c32a80 >>>>>>>>>>>>>>> /var/lib/ceph/osd/ceph-1/block) close >>>>>>>>>>>>>>> 2018-10-01 18:02:41.443 7fc9226c6240 -1 osd.1 0 OSD:init: >>>>>>>>>>>>>>> unable to mount object store >>>>>>>>>>>>>>> 2018-10-01 18:02:41.443 7fc9226c6240 -1 ** ERROR: osd init >>>>>>>>>>>>>>> failed: (5) Input/output error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 1.10.2018, at 18:09, Igor Fedotov <ifedo...@suse.de> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Well, actually you can avoid bluestore-tool rebuild. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You'll need to edit the first chunk of blocks.db where labels >>>>>>>>>>>>>>>> are stored. (Please make a backup first!!!) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Size label is stored at offset 0x52 and is 8 bytes long - >>>>>>>>>>>>>>>> little-endian 64bit integer encoding. (Please verify that old >>>>>>>>>>>>>>>> value at this offset exactly corresponds to you original >>>>>>>>>>>>>>>> volume size and/or 'size' label reported by >>>>>>>>>>>>>>>> ceph-bluestore-tool). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So you have to put new DB volume size there. Or you can send >>>>>>>>>>>>>>>> the first 4K chunk (e.g. extracted with dd) along with new DB >>>>>>>>>>>>>>>> volume size (in bytes) to me and I'll do that for you. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Igor >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 10/1/2018 5:32 PM, Igor Fedotov wrote: >>>>>>>>>>>>>>>>> On 10/1/2018 5:03 PM, Sergey Malinin wrote: >>>>>>>>>>>>>>>>>> Before I received your response, I had already added 20GB to >>>>>>>>>>>>>>>>>> the OSD (by epanding LV followed by bluefs-bdev-expand) and >>>>>>>>>>>>>>>>>> ran "ceph-kvstore-tool bluestore-kv <path> compact", however >>>>>>>>>>>>>>>>>> it still needs more space. >>>>>>>>>>>>>>>>>> Is that because I didn't update DB size with set-label-key? >>>>>>>>>>>>>>>>> In mimic you need to run both "bluefs-bdev-expand" and >>>>>>>>>>>>>>>>> "set-label-key" command to commit bluefs volume expansion. >>>>>>>>>>>>>>>>> Unfortunately the last command doesn't handle "size" label >>>>>>>>>>>>>>>>> properly. That's why you might need to backport and rebuild >>>>>>>>>>>>>>>>> with the mentioned commits. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What exactly is the label-key that needs to be updated, as I >>>>>>>>>>>>>>>>>> couldn't find which one is related to DB: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> # ceph-bluestore-tool show-label --path >>>>>>>>>>>>>>>>>> /var/lib/ceph/osd/ceph-1 >>>>>>>>>>>>>>>>>> inferring bluefs devices from bluestore path >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> "/var/lib/ceph/osd/ceph-1/block": { >>>>>>>>>>>>>>>>>> "osd_uuid": "f8f122ee-70a6-4c54-8eb0-9b42205b1ecc", >>>>>>>>>>>>>>>>>> "size": 471305551872, >>>>>>>>>>>>>>>>>> "btime": "2018-07-31 03:06:43.751243", >>>>>>>>>>>>>>>>>> "description": "main", >>>>>>>>>>>>>>>>>> "bluefs": "1", >>>>>>>>>>>>>>>>>> "ceph_fsid": >>>>>>>>>>>>>>>>>> "7d320499-5b3f-453e-831f-60d4db9a4533", >>>>>>>>>>>>>>>>>> "kv_backend": "rocksdb", >>>>>>>>>>>>>>>>>> "magic": "ceph osd volume v026", >>>>>>>>>>>>>>>>>> "mkfs_done": "yes", >>>>>>>>>>>>>>>>>> "osd_key": "XXX", >>>>>>>>>>>>>>>>>> "ready": "ready", >>>>>>>>>>>>>>>>>> "whoami": "1" >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> 'size' label but your output is for block(aka slow) device. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It should return labels for db/wal devices as well (block.db >>>>>>>>>>>>>>>>> and block.wal symlinks respectively). It works for me in >>>>>>>>>>>>>>>>> master, can't verify with mimic at the moment though.. >>>>>>>>>>>>>>>>> Here is output for master: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # bin/ceph-bluestore-tool show-label --path dev/osd0 >>>>>>>>>>>>>>>>> inferring bluefs devices from bluestore path >>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>> "dev/osd0/block": { >>>>>>>>>>>>>>>>> "osd_uuid": "404dcbe9-3f8d-4ef5-ac59-2582454a9a75", >>>>>>>>>>>>>>>>> "size": 21474836480, >>>>>>>>>>>>>>>>> "btime": "2018-09-10 15:55:09.044039", >>>>>>>>>>>>>>>>> "description": "main", >>>>>>>>>>>>>>>>> "bluefs": "1", >>>>>>>>>>>>>>>>> "ceph_fsid": "56eddc15-11b9-4e0b-9192-e391fbae551c", >>>>>>>>>>>>>>>>> "kv_backend": "rocksdb", >>>>>>>>>>>>>>>>> "magic": "ceph osd volume v026", >>>>>>>>>>>>>>>>> "mkfs_done": "yes", >>>>>>>>>>>>>>>>> "osd_key": >>>>>>>>>>>>>>>>> "AQCsaZZbYTxXJBAAe3jJI4p6WbMjvA8CBBUJbA==", >>>>>>>>>>>>>>>>> "ready": "ready", >>>>>>>>>>>>>>>>> "whoami": "0" >>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>> "dev/osd0/block.wal": { >>>>>>>>>>>>>>>>> "osd_uuid": "404dcbe9-3f8d-4ef5-ac59-2582454a9a75", >>>>>>>>>>>>>>>>> "size": 1048576000, >>>>>>>>>>>>>>>>> "btime": "2018-09-10 15:55:09.044985", >>>>>>>>>>>>>>>>> "description": "bluefs wal" >>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>> "dev/osd0/block.db": { >>>>>>>>>>>>>>>>> "osd_uuid": "404dcbe9-3f8d-4ef5-ac59-2582454a9a75", >>>>>>>>>>>>>>>>> "size": 1048576000, >>>>>>>>>>>>>>>>> "btime": "2018-09-10 15:55:09.044469", >>>>>>>>>>>>>>>>> "description": "bluefs db" >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You can try --dev option instead of --path, e.g. >>>>>>>>>>>>>>>>> ceph-bluestore-tool show-label --dev <path-to-block.db> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 1.10.2018, at 16:48, Igor Fedotov <ifedo...@suse.de> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> This looks like a sort of deadlock when BlueFS needs some >>>>>>>>>>>>>>>>>>> additional space to replay the log left after the crash. >>>>>>>>>>>>>>>>>>> Which happens during BlueFS open. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> But such a space (at slow device as DB is full) is gifted >>>>>>>>>>>>>>>>>>> in background during bluefs rebalance procedure which will >>>>>>>>>>>>>>>>>>> occur after the open. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hence OSDs stuck in permanent crashing.. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The only way to recover I can suggest for now is to expand >>>>>>>>>>>>>>>>>>> DB volumes. You can do that with lvm tools if you have any >>>>>>>>>>>>>>>>>>> spare space for that. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Once resized you'll need ceph-bluestore-tool to indicate >>>>>>>>>>>>>>>>>>> volume expansion to BlueFS (bluefs-bdev-expand command ) >>>>>>>>>>>>>>>>>>> and finally update DB volume size label with set-label-key >>>>>>>>>>>>>>>>>>> command. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The latter is a bit tricky for mimic - you might need to >>>>>>>>>>>>>>>>>>> backport >>>>>>>>>>>>>>>>>>> https://github.com/ceph/ceph/pull/22085/commits/ffac450da5d6e09cf14b8363b35f21819b48f38b >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> and rebuild ceph-bluestore-tool. Alternatively you can >>>>>>>>>>>>>>>>>>> backport >>>>>>>>>>>>>>>>>>> https://github.com/ceph/ceph/pull/22085/commits/71c3b58da4e7ced3422bce2b1da0e3fa9331530b >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> then bluefs expansion and label updates will occur in a >>>>>>>>>>>>>>>>>>> single step. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I'll do these backports in upstream but this will take some >>>>>>>>>>>>>>>>>>> time to pass all the procedures and get into official mimic >>>>>>>>>>>>>>>>>>> release. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Will fire a ticket to fix the original issue as well. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Igor >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 10/1/2018 3:28 PM, Sergey Malinin wrote: >>>>>>>>>>>>>>>>>>>> These are LVM bluestore NVMe SSDs created with >>>>>>>>>>>>>>>>>>>> "ceph-volume --lvm prepare --bluestore /dev/nvme0n1p3" >>>>>>>>>>>>>>>>>>>> i.e. without specifying wal/db devices. >>>>>>>>>>>>>>>>>>>> OSDs were created with bluestore_min_alloc_size_ssd=4096, >>>>>>>>>>>>>>>>>>>> another modified setting is >>>>>>>>>>>>>>>>>>>> bluestore_cache_kv_max=1073741824 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> DB/block usage collected by prometheus module for 3 failed >>>>>>>>>>>>>>>>>>>> and 1 survived OSDs: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_total_bytes{ceph_daemon="osd.0"} >>>>>>>>>>>>>>>>>>>> 65493008384.0 >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_total_bytes{ceph_daemon="osd.1"} >>>>>>>>>>>>>>>>>>>> 49013587968.0 >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_total_bytes{ceph_daemon="osd.2"} >>>>>>>>>>>>>>>>>>>> 76834406400.0 --> this one has survived >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_total_bytes{ceph_daemon="osd.3"} >>>>>>>>>>>>>>>>>>>> 63726157824.0 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_used_bytes{ceph_daemon="osd.0"} >>>>>>>>>>>>>>>>>>>> 65217232896.0 >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_used_bytes{ceph_daemon="osd.1"} >>>>>>>>>>>>>>>>>>>> 48944381952.0 >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_used_bytes{ceph_daemon="osd.2"} >>>>>>>>>>>>>>>>>>>> 68093476864.0 >>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_used_bytes{ceph_daemon="osd.3"} >>>>>>>>>>>>>>>>>>>> 63632834560.0 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes{ceph_daemon="osd.0"} 471305551872.0 >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes{ceph_daemon="osd.1"} 471305551872.0 >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes{ceph_daemon="osd.2"} 471305551872.0 >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes{ceph_daemon="osd.3"} 471305551872.0 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes_used{ceph_daemon="osd.0"} >>>>>>>>>>>>>>>>>>>> 222328213504.0 >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes_used{ceph_daemon="osd.1"} >>>>>>>>>>>>>>>>>>>> 214472544256.0 >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes_used{ceph_daemon="osd.2"} >>>>>>>>>>>>>>>>>>>> 163603996672.0 >>>>>>>>>>>>>>>>>>>> ceph_osd_stat_bytes_used{ceph_daemon="osd.3"} >>>>>>>>>>>>>>>>>>>> 212806815744.0 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> First crashed OSD was doing DB compaction, others crashed >>>>>>>>>>>>>>>>>>>> shortly after during backfilling. Workload was >>>>>>>>>>>>>>>>>>>> "ceph-data-scan scan_inodes" filling metadata pool located >>>>>>>>>>>>>>>>>>>> on these OSDs at the rate close to 10k objects/second. >>>>>>>>>>>>>>>>>>>> Here is the log excerpt of the first crash occurrence: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:12.762 7fbf16dd6700 0 >>>>>>>>>>>>>>>>>>>> bluestore(/var/lib/ceph/osd/ceph-1) >>>>>>>>>>>>>>>>>>>> _balance_bluefs_freespace no allocate on 0x80000000 >>>>>>>>>>>>>>>>>>>> min_alloc_size 0x1000 >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:12.886 7fbf1e5e5700 4 rocksdb: >>>>>>>>>>>>>>>>>>>> [/build/ceph-13.2.2/src/rocksdb/db/compaction_job.cc:1166] >>>>>>>>>>>>>>>>>>>> [default] [JOB 24] Generated table #89741: 106356 keys, >>>>>>>>>>>>>>>>>>>> 68110589 bytes >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:12.886 7fbf1e5e5700 4 rocksdb: >>>>>>>>>>>>>>>>>>>> EVENT_LOG_v1 {"time_micros": 1538353632892744, "cf_name": >>>>>>>>>>>>>>>>>>>> "default", "job": 24, "event": "table_file_creation", >>>>>>>>>>>>>>>>>>>> "file_number": 89741, "file_size": 68110589, >>>>>>>>>>>>>>>>>>>> "table_properties": {"data_size": 67112903, "index_size": >>>>>>>>>>>>>>>>>>>> 579319, "filter_size": 417316, "raw_key_size": 6733561, >>>>>>>>>>>>>>>>>>>> "raw_average_key_size": 63, "raw_value_size": 60994583, >>>>>>>>>>>>>>>>>>>> "raw_average_value_size": 573, "num_data_blocks": 16336, >>>>>>>>>>>>>>>>>>>> "num_entries": 106356, "filter_policy_name": >>>>>>>>>>>>>>>>>>>> "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "14444", >>>>>>>>>>>>>>>>>>>> "kMergeOperands": "0"}} >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:12.934 7fbf1e5e5700 4 rocksdb: >>>>>>>>>>>>>>>>>>>> [/build/ceph-13.2.2/src/rocksdb/db/compaction_job.cc:1166] >>>>>>>>>>>>>>>>>>>> [default] [JOB 24] Generated table #89742: 23214 keys, >>>>>>>>>>>>>>>>>>>> 16352315 bytes >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:12.934 7fbf1e5e5700 4 rocksdb: >>>>>>>>>>>>>>>>>>>> EVENT_LOG_v1 {"time_micros": 1538353632938670, "cf_name": >>>>>>>>>>>>>>>>>>>> "default", "job": 24, "event": "table_file_creation", >>>>>>>>>>>>>>>>>>>> "file_number": 89742, "file_size": 16352315, >>>>>>>>>>>>>>>>>>>> "table_properties": {"data_size": 16116986, "index_size": >>>>>>>>>>>>>>>>>>>> 139894, "filter_size": 94386, "raw_key_size": 1470883, >>>>>>>>>>>>>>>>>>>> "raw_average_key_size": 63, "raw_value_size": 14775006, >>>>>>>>>>>>>>>>>>>> "raw_average_value_size": 636, "num_data_blocks": 3928, >>>>>>>>>>>>>>>>>>>> "num_entries": 23214, "filter_policy_name": >>>>>>>>>>>>>>>>>>>> "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "90", >>>>>>>>>>>>>>>>>>>> "kMergeOperands": "0"}} >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:13.042 7fbf1e5e5700 1 bluefs _allocate >>>>>>>>>>>>>>>>>>>> failed to allocate 0x4100000 on bdev 1, free 0x1a00000; >>>>>>>>>>>>>>>>>>>> fallback to bdev 2 >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:13.042 7fbf1e5e5700 -1 bluefs _allocate >>>>>>>>>>>>>>>>>>>> failed to allocate 0x4100000 on bdev 2, dne >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:13.042 7fbf1e5e5700 -1 bluefs >>>>>>>>>>>>>>>>>>>> _flush_range allocated: 0x0 offset: 0x0 length: 0x40ea9f1 >>>>>>>>>>>>>>>>>>>> 2018-10-01 03:27:13.046 7fbf1e5e5700 -1 >>>>>>>>>>>>>>>>>>>> /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: In function >>>>>>>>>>>>>>>>>>>> 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, >>>>>>>>>>>>>>>>>>>> uint64_t)' thread 7fbf1e5e5700 time 2018-10-01 >>>>>>>>>>>>>>>>>>>> 03:27:13.048298 >>>>>>>>>>>>>>>>>>>> /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: 1663: >>>>>>>>>>>>>>>>>>>> FAILED assert(0 == "bluefs enospc") >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ceph version 13.2.2 >>>>>>>>>>>>>>>>>>>> (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) >>>>>>>>>>>>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, >>>>>>>>>>>>>>>>>>>> int, char const*)+0x102) [0x7fbf2d4fe5c2] >>>>>>>>>>>>>>>>>>>> 2: (()+0x26c787) [0x7fbf2d4fe787] >>>>>>>>>>>>>>>>>>>> 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned >>>>>>>>>>>>>>>>>>>> long, unsigned long)+0x1ab4) [0x5619325114b4] >>>>>>>>>>>>>>>>>>>> 4: (BlueRocksWritableFile::Flush()+0x3d) >>>>>>>>>>>>>>>>>>>> [0x561932527c1d] >>>>>>>>>>>>>>>>>>>> 5: (rocksdb::WritableFileWriter::Flush()+0x1b9) >>>>>>>>>>>>>>>>>>>> [0x56193271c399] >>>>>>>>>>>>>>>>>>>> 6: (rocksdb::WritableFileWriter::Sync(bool)+0x3b) >>>>>>>>>>>>>>>>>>>> [0x56193271d42b] >>>>>>>>>>>>>>>>>>>> 7: >>>>>>>>>>>>>>>>>>>> (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status >>>>>>>>>>>>>>>>>>>> const&, rocksdb::CompactionJob::SubcompactionState*, >>>>>>>>>>>>>>>>>>>> rocksdb::RangeDelAggregator*, CompactionIterationStats*, >>>>>>>>>>>>>>>>>>>> rocksdb::Slice const*)+0x3db) [0x56193276098b] >>>>>>>>>>>>>>>>>>>> 8: >>>>>>>>>>>>>>>>>>>> (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x7d9) >>>>>>>>>>>>>>>>>>>> [0x561932763da9] >>>>>>>>>>>>>>>>>>>> 9: (rocksdb::CompactionJob::Run()+0x314) >>>>>>>>>>>>>>>>>>>> [0x561932765504] >>>>>>>>>>>>>>>>>>>> 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, >>>>>>>>>>>>>>>>>>>> rocksdb::JobContext*, rocksdb::LogBuffer*, >>>>>>>>>>>>>>>>>>>> rocksdb::DBImpl::PrepickedCompaction*)+0xc54) >>>>>>>>>>>>>>>>>>>> [0x5619325b5c44] >>>>>>>>>>>>>>>>>>>> 11: >>>>>>>>>>>>>>>>>>>> (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, >>>>>>>>>>>>>>>>>>>> rocksdb::Env::Priority)+0x397) [0x5619325b8557] >>>>>>>>>>>>>>>>>>>> 12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) >>>>>>>>>>>>>>>>>>>> [0x5619325b8cd7] >>>>>>>>>>>>>>>>>>>> 13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned >>>>>>>>>>>>>>>>>>>> long)+0x266) [0x5619327a5e36] >>>>>>>>>>>>>>>>>>>> 14: >>>>>>>>>>>>>>>>>>>> (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) >>>>>>>>>>>>>>>>>>>> [0x5619327a5fb7] >>>>>>>>>>>>>>>>>>>> 15: (()+0xbe733) [0x7fbf2b500733] >>>>>>>>>>>>>>>>>>>> 16: (()+0x76db) [0x7fbf2bbf86db] >>>>>>>>>>>>>>>>>>>> 17: (clone()+0x3f) [0x7fbf2abbc88f] >>>>>>>>>>>>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS >>>>>>>>>>>>>>>>>>>> <executable>` is needed to interpret this. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 1.10.2018, at 15:01, Igor Fedotov <ifedo...@suse.de> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Sergey, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> could you please provide more details on your OSDs ? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> What are sizes for DB/block devices? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Do you have any modifications in BlueStore config >>>>>>>>>>>>>>>>>>>>> settings? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Can you share stats you're referring to? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Igor >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 10/1/2018 12:29 PM, Sergey Malinin wrote: >>>>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>>>> 3 of 4 NVME OSDs crashed at the same time on assert(0 == >>>>>>>>>>>>>>>>>>>>>> "bluefs enospc") and no longer start. >>>>>>>>>>>>>>>>>>>>>> Stats collected just before crash show that >>>>>>>>>>>>>>>>>>>>>> ceph_bluefs_db_used_bytes is 100% used. Although OSDs >>>>>>>>>>>>>>>>>>>>>> have over 50% of free space, it is not reallocated for >>>>>>>>>>>>>>>>>>>>>> DB usage. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2018-10-01 12:18:06.744 7f1d6a04d240 1 bluefs _allocate >>>>>>>>>>>>>>>>>>>>>> failed to allocate 0x100000 on bdev 1, free 0x0; >>>>>>>>>>>>>>>>>>>>>> fallback to bdev 2 >>>>>>>>>>>>>>>>>>>>>> 2018-10-01 12:18:06.744 7f1d6a04d240 -1 bluefs _allocate >>>>>>>>>>>>>>>>>>>>>> failed to allocate 0x100000 on bdev 2, dne >>>>>>>>>>>>>>>>>>>>>> 2018-10-01 12:18:06.744 7f1d6a04d240 -1 bluefs >>>>>>>>>>>>>>>>>>>>>> _flush_range allocated: 0x0 offset: 0x0 length: 0xa8700 >>>>>>>>>>>>>>>>>>>>>> 2018-10-01 12:18:06.748 7f1d6a04d240 -1 >>>>>>>>>>>>>>>>>>>>>> /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: In >>>>>>>>>>>>>>>>>>>>>> function 'int BlueFS::_flush_range(BlueFS::FileWriter*, >>>>>>>>>>>>>>>>>>>>>> uint64_t, uint64_t)' thread 7f1d6a04d240 time 2018-10-01 >>>>>>>>>>>>>>>>>>>>>> 12:18:06.746800 >>>>>>>>>>>>>>>>>>>>>> /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: 1663: >>>>>>>>>>>>>>>>>>>>>> FAILED assert(0 == "bluefs enospc") >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> ceph version 13.2.2 >>>>>>>>>>>>>>>>>>>>>> (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) >>>>>>>>>>>>>>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char >>>>>>>>>>>>>>>>>>>>>> const*, int, char const*)+0x102) [0x7f1d6146f5c2] >>>>>>>>>>>>>>>>>>>>>> 2: (()+0x26c787) [0x7f1d6146f787] >>>>>>>>>>>>>>>>>>>>>> 3: (BlueFS::_flush_range(BlueFS::FileWriter*, >>>>>>>>>>>>>>>>>>>>>> unsigned long, unsigned long)+0x1ab4) [0x5586b22684b4] >>>>>>>>>>>>>>>>>>>>>> 4: (BlueRocksWritableFile::Flush()+0x3d) >>>>>>>>>>>>>>>>>>>>>> [0x5586b227ec1d] >>>>>>>>>>>>>>>>>>>>>> 5: (rocksdb::WritableFileWriter::Flush()+0x1b9) >>>>>>>>>>>>>>>>>>>>>> [0x5586b2473399] >>>>>>>>>>>>>>>>>>>>>> 6: (rocksdb::WritableFileWriter::Sync(bool)+0x3b) >>>>>>>>>>>>>>>>>>>>>> [0x5586b247442b] >>>>>>>>>>>>>>>>>>>>>> 7: >>>>>>>>>>>>>>>>>>>>>> (rocksdb::BuildTable(std::__cxx11::basic_string<char, >>>>>>>>>>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> > const&, >>>>>>>>>>>>>>>>>>>>>> rocksdb::Env*, rocksdb::ImmutableCFOptions const&, >>>>>>>>>>>>>>>>>>>>>> rocksdb::MutableCFOptions const&, rocksdb::EnvOptions >>>>>>>>>>>>>>>>>>>>>> const&, rock >>>>>>>>>>>>>>>>>>>>>> sdb::TableCache*, rocksdb::InternalIterator*, >>>>>>>>>>>>>>>>>>>>>> std::unique_ptr<rocksdb::InternalIterator, >>>>>>>>>>>>>>>>>>>>>> std::default_delete<rocksdb::InternalIterator> >, >>>>>>>>>>>>>>>>>>>>>> rocksdb::FileMetaData*, rocksdb::InternalKeyComparator >>>>>>>>>>>>>>>>>>>>>> const&, std::vector<std::unique_ptr< >>>>>>>>>>>>>>>>>>>>>> rocksdb::IntTblPropCollectorFactory, >>>>>>>>>>>>>>>>>>>>>> std::default_delete<rocksdb::IntTblPropCollectorFactory> >>>>>>>>>>>>>>>>>>>>>> >, >>>>>>>>>>>>>>>>>>>>>> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> std::default_delete<rocksdb::IntTblPropCollectorFactory> >>>>>>>>>>>>>>>>>>>>>> > > > co >>>>>>>>>>>>>>>>>>>>>> nst*, unsigned int, std::__cxx11::basic_string<char, >>>>>>>>>>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> > const&, >>>>>>>>>>>>>>>>>>>>>> std::vector<unsigned long, std::allocator<unsigned long> >>>>>>>>>>>>>>>>>>>>>> >, unsigned long, rocksdb::SnapshotChecker*, >>>>>>>>>>>>>>>>>>>>>> rocksdb::Compression >>>>>>>>>>>>>>>>>>>>>> Type, rocksdb::CompressionOptions const&, bool, >>>>>>>>>>>>>>>>>>>>>> rocksdb::InternalStats*, >>>>>>>>>>>>>>>>>>>>>> rocksdb::TableFileCreationReason, rocksdb::EventLogger*, >>>>>>>>>>>>>>>>>>>>>> int, rocksdb::Env::IOPriority, >>>>>>>>>>>>>>>>>>>>>> rocksdb::TableProperties*, int, unsigned long, unsigned >>>>>>>>>>>>>>>>>>>>>> long, rocksdb >>>>>>>>>>>>>>>>>>>>>> ::Env::WriteLifeTimeHint)+0x1e24) [0x5586b249ef94] >>>>>>>>>>>>>>>>>>>>>> 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, >>>>>>>>>>>>>>>>>>>>>> rocksdb::ColumnFamilyData*, rocksdb::MemTable*, >>>>>>>>>>>>>>>>>>>>>> rocksdb::VersionEdit*)+0xcb7) [0x5586b2321457] >>>>>>>>>>>>>>>>>>>>>> 9: >>>>>>>>>>>>>>>>>>>>>> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned >>>>>>>>>>>>>>>>>>>>>> long, std::allocator<unsigned long> > const&, unsigned >>>>>>>>>>>>>>>>>>>>>> long*, bool)+0x19de) [0x5586b232373e] >>>>>>>>>>>>>>>>>>>>>> 10: >>>>>>>>>>>>>>>>>>>>>> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, >>>>>>>>>>>>>>>>>>>>>> std::allocator<rocksdb::ColumnFamilyDescriptor> > >>>>>>>>>>>>>>>>>>>>>> const&, bool, bool, bool)+0x5d4) [0x5586b23242f4] >>>>>>>>>>>>>>>>>>>>>> 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, >>>>>>>>>>>>>>>>>>>>>> std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>>>>>>>>>>>>>>>>> std::allocator<char> > const&, >>>>>>>>>>>>>>>>>>>>>> std::vector<rocksdb::ColumnFamilyDescriptor, >>>>>>>>>>>>>>>>>>>>>> std::allocator<rocksdb::ColumnFamilyDescri >>>>>>>>>>>>>>>>>>>>>> ptor> > const&, >>>>>>>>>>>>>>>>>>>>>> std::vector<rocksdb::ColumnFamilyHandle*, >>>>>>>>>>>>>>>>>>>>>> std::allocator<rocksdb::ColumnFamilyHandle*> >*, >>>>>>>>>>>>>>>>>>>>>> rocksdb::DB**, bool)+0x68b) [0x5586b232559b] >>>>>>>>>>>>>>>>>>>>>> 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, >>>>>>>>>>>>>>>>>>>>>> std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>>>>>>>>>>>>>>>>> std::allocator<char> > const&, >>>>>>>>>>>>>>>>>>>>>> std::vector<rocksdb::ColumnFamilyDescriptor, >>>>>>>>>>>>>>>>>>>>>> std::allocator<rocksdb::ColumnFamilyDescriptor >>>>>>>>>>>>>>>>>>>>>>>> const&, std::vector<rocksdb::ColumnFamilyHandle*, >>>>>>>>>>>>>>>>>>>>>>>> std::allocator<rocksdb::ColumnFamilyHandle*> >*, >>>>>>>>>>>>>>>>>>>>>>>> rocksdb::DB**)+0x22) [0x5586b2326e72] >>>>>>>>>>>>>>>>>>>>>> 13: (RocksDBStore::do_open(std::ostream&, bool, >>>>>>>>>>>>>>>>>>>>>> std::vector<KeyValueDB::ColumnFamily, >>>>>>>>>>>>>>>>>>>>>> std::allocator<KeyValueDB::ColumnFamily> > >>>>>>>>>>>>>>>>>>>>>> const*)+0x170c) [0x5586b220219c] >>>>>>>>>>>>>>>>>>>>>> 14: (BlueStore::_open_db(bool, bool)+0xd8e) >>>>>>>>>>>>>>>>>>>>>> [0x5586b218ee1e] >>>>>>>>>>>>>>>>>>>>>> 15: (BlueStore::_mount(bool, bool)+0x4b7) >>>>>>>>>>>>>>>>>>>>>> [0x5586b21bf807] >>>>>>>>>>>>>>>>>>>>>> 16: (OSD::init()+0x295) [0x5586b1d673c5] >>>>>>>>>>>>>>>>>>>>>> 17: (main()+0x268d) [0x5586b1c554ed] >>>>>>>>>>>>>>>>>>>>>> 18: (__libc_start_main()+0xe7) [0x7f1d5ea2db97] >>>>>>>>>>>>>>>>>>>>>> 19: (_start()+0x2a) [0x5586b1d1d7fa] >>>>>>>>>>>>>>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS >>>>>>>>>>>>>>>>>>>>>> <executable>` is needed to interpret this. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>> ceph-users mailing list >>>>>>>>>>>>>>>>>>>>>> ceph-users@lists.ceph.com >>>>>>>>>>>>>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> ceph-users mailing list >>>>>>>>>>>>>>>>> ceph-users@lists.ceph.com >>>>>>>>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list >>>>>>>>>>>> ceph-users@lists.ceph.com >>>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
