SOLVED
Short: I followed the procedure to replace an OSD.
Long: I reweighted the flapping OSD to 0 until it was done, then marked it
out, and then unmounted it, then followed the replacement procedure[1],
then restore weight. I attempted to recreate my actions from bash history
from various terminals. Please excuse any mistakes:
admin:$ X=0 # ID of OSD
admin:$ ceph osd crush reweight osd.$X 0.0
admin:$ ceph -w # observe data migration
admin:$ ceph osd out $X
osd:$ systemctl stop ceph.target
admin:$ ceph osd destroy $X --yes-i-really-mean-it
osd:$ umount /dev/sdb1
osd:$ ceph-disk zap /dev/sdb
osd:$ ceph-disk prepare --bluestore /dev/sdb --osd-id 0 --osd-uuid
`uuidgen`
osd:$ ceph-disk activate /dev/sdb1
admin:$ ceph osd crush reweight osd.0 9.09560
admin:$ ceph -w # observe data migration
# I also found it helpful to set 'nodown' for several minutes because two
of the OSDs kept marking eachother down.
admin:$ ceph osd set nodown
# Once all was 'active+clean', I removed the 'nodown' flag.
admin:$ ceph osd unset nodown
And for the first time since upgrading from Kraken to all the Luminous
releases through 12.1.2, my cluster is finally HEALTH_OK. Yay!
roger@desktop:~/ceph-cluster$ ceph -w
cluster:
id: eea7b78c-b138-40fc-9f3e-3d77afb770f0
health: HEALTH_OK
services:
mon: 3 daemons, quorum nuc1,nuc2,nuc3
mgr: nuc3(active), standbys: nuc1, nuc2
osd: 3 osds: 3 up, 3 in
rgw: 1 daemon active
data:
pools: 19 pools, 372 pgs
objects: 54278 objects, 71724 MB
usage: 121 GB used, 27820 GB / 27941 GB avail
pgs: 372 active+clean
1.
http://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/#replacing-an-osd
On Wed, Aug 2, 2017 at 11:08 AM Roger Brown <rogerpbr...@gmail.com> wrote:
> Hi,
>
> My OSD's were continuously crashing in cephx_verify_authorizer() while on
> Luminous v12.1.0 and v12.1.1, but the crashes stopped once I upgraded to
> v12.1.2.
>
> Now however, one of my OSDs is continuing to crash. Looking closer, the
> crash reason is different reason and started with v12.1.1.
>
> I've been troubleshooting with the aid of
> http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/
> .
>
> I'm considering reweight to 0 and then redeploy that OSD from scratch,
> unless you can do a filesystem repair on bluestore/rocksdb somehow. Please
> advise.
>
>
> Data follows...
>
> Log:
> roger@osd3:~$ sudo journalctl -u ceph-osd@0 --no-pager
> ...
> Aug 02 10:38:47 osd3 systemd[1]: ceph-osd@0.service: Failed with result
> 'signal'.
> Aug 02 10:39:07 osd3 systemd[1]: ceph-osd@0.service: Service hold-off
> time over, scheduling restart.
> Aug 02 10:39:07 osd3 systemd[1]: Stopped Ceph object storage daemon osd.0.
> Aug 02 10:39:07 osd3 systemd[1]: Starting Ceph object storage daemon
> osd.0...
> Aug 02 10:39:07 osd3 systemd[1]: Started Ceph object storage daemon osd.0.
> Aug 02 10:39:07 osd3 ceph-osd[7413]: starting osd.0 at - osd_data
> /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
> Aug 02 10:40:48 osd3 ceph-osd[7413]: 2017-08-02 10:40:48.583063
> 7f5262cc3e00 -1 osd.0 25924 log_to_monitors {default=true}
> Aug 02 10:43:32 osd3 ceph-osd[7413]: *** Caught signal (Aborted) **
> Aug 02 10:43:32 osd3 ceph-osd[7413]: in thread 7f524861b700
> thread_name:tp_osd_tp
> Aug 02 10:43:32 osd3 ceph-osd[7413]: ceph version 12.1.2
> (b661348f156f148d764b998b65b90451f096cb27) luminous (rc)
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 1: (()+0xa9a964) [0x5623f0a9c964]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 2: (()+0x11390) [0x7f52611a6390]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 3: (pread64()+0x33) [0x7f52611a5d43]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 4:
> (KernelDevice::direct_read_unaligned(unsigned long, unsigned long,
> char*)+0x81) [0x5623f0a7cfc1]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 5:
> (KernelDevice::read_random(unsigned long, unsigned long, char*,
> bool)+0x4f3) [0x5623f0a7da43]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 6:
> (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long,
> char*)+0x4fa) [0x5623f0a4d9ca]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 7:
> (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
> rocksdb::Slice*, char*) const+0x20) [0x5623f0a77e10]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 8:
> (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
> rocksdb::Slice*, char*) const+0xf8f) [0x5623f0e50acf]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 9:
> (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
> rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle
> const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool,
> rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x5f3)
> [0x5623f0e21383]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 10: (()+0xe0f7c6) [0x5623f0e117c6]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 11:
> (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*,
> rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice,
> rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x2f8)
> [0x5623f0e13928]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 12:
> (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*,
> rocksdb::ReadOptions const&, rocksdb::BlockHandle const&,
> rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x5623f0e13d2c]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 13:
> (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice
> const&)+0x97) [0x5623f0e1c4a7]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 14: (()+0xe4576e) [0x5623f0e4776e]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 15: (()+0xe45836) [0x5623f0e47836]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 16: (()+0xe459b1) [0x5623f0e479b1]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 17:
> (rocksdb::MergingIterator::Next()+0x449) [0x5623f0e2ab09]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 18:
> (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x182)
> [0x5623f0ec7ed2]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 19: (rocksdb::DBIter::Next()+0x1eb)
> [0x5623f0ec8c8b]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 20:
> (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x5623f09dd58a]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 21:
> (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&,
> ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t>
> >*, ghobject_t*)+0x1170) [0x5623f093d250]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 22:
> (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
> ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t,
> std::allocator<ghobject_t> >*, ghobject_t*)+0x25a) [0x5623f093e6ea]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 23:
> (PGBackend::objects_list_range(hobject_t const&, hobject_t const&,
> snapid_t, std::vector<hobject_t, std::allocator<hobject_t> >*,
> std::vector<ghobject_t, std::allocator<ghobject_t> >*)+0x192)
> [0x5623f0700ef2]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 24:
> (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool, unsigned
> int, ThreadPool::TPHandle&)+0x200) [0x5623f05a8b30]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 25:
> (PG::chunky_scrub(ThreadPool::TPHandle&)+0x3ea) [0x5623f05d61ca]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 26: (PG::scrub(unsigned int,
> ThreadPool::TPHandle&)+0x45c) [0x5623f05d7cec]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 27:
> (OSD::ShardedOpWQ::_process(unsigned int,
> ceph::heartbeat_handle_d*)+0x12d0) [0x5623f05179e0]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 28:
> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884)
> [0x5623f0ae44e4]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 29:
> (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5623f0ae7520]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 30: (()+0x76ba) [0x7f526119c6ba]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 31: (clone()+0x6d) [0x7f52602133dd]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 2017-08-02 10:43:32.251841
> 7f524861b700 -1 *** Caught signal (Aborted) **
> Aug 02 10:43:32 osd3 ceph-osd[7413]: in thread 7f524861b700
> thread_name:tp_osd_tp
> Aug 02 10:43:32 osd3 ceph-osd[7413]: ceph version 12.1.2
> (b661348f156f148d764b998b65b90451f096cb27) luminous (rc)
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 1: (()+0xa9a964) [0x5623f0a9c964]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 2: (()+0x11390) [0x7f52611a6390]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 3: (pread64()+0x33) [0x7f52611a5d43]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 4:
> (KernelDevice::direct_read_unaligned(unsigned long, unsigned long,
> char*)+0x81) [0x5623f0a7cfc1]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 5:
> (KernelDevice::read_random(unsigned long, unsigned long, char*,
> bool)+0x4f3) [0x5623f0a7da43]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 6:
> (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long,
> char*)+0x4fa) [0x5623f0a4d9ca]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 7:
> (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
> rocksdb::Slice*, char*) const+0x20) [0x5623f0a77e10]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 8:
> (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
> rocksdb::Slice*, char*) const+0xf8f) [0x5623f0e50acf]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 9:
> (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
> rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle
> const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool,
> rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x5f3)
> [0x5623f0e21383]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 10: (()+0xe0f7c6) [0x5623f0e117c6]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 11:
> (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*,
> rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice,
> rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x2f8)
> [0x5623f0e13928]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 12:
> (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*,
> rocksdb::ReadOptions const&, rocksdb::BlockHandle const&,
> rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x5623f0e13d2c]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 13:
> (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice
> const&)+0x97) [0x5623f0e1c4a7]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 14: (()+0xe4576e) [0x5623f0e4776e]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 15: (()+0xe45836) [0x5623f0e47836]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 16: (()+0xe459b1) [0x5623f0e479b1]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 17:
> (rocksdb::MergingIterator::Next()+0x449) [0x5623f0e2ab09]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 18:
> (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x182)
> [0x5623f0ec7ed2]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 19: (rocksdb::DBIter::Next()+0x1eb)
> [0x5623f0ec8c8b]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 20:
> (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x5623f09dd58a]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 21:
> (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&,
> ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t>
> >*, ghobject_t*)+0x1170) [0x5623f093d250]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 22:
> (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
> ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t,
> std::allocator<ghobject_t> >*, ghobject_t*)+0x25a) [0x5623f093e6ea]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 23:
> (PGBackend::objects_list_range(hobject_t const&, hobject_t const&,
> snapid_t, std::vector<hobject_t, std::allocator<hobject_t> >*,
> std::vector<ghobject_t, std::allocator<ghobject_t> >*)+0x192)
> [0x5623f0700ef2]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 24:
> (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool, unsigned
> int, ThreadPool::TPHandle&)+0x200) [0x5623f05a8b30]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 25:
> (PG::chunky_scrub(ThreadPool::TPHandle&)+0x3ea) [0x5623f05d61ca]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 26: (PG::scrub(unsigned int,
> ThreadPool::TPHandle&)+0x45c) [0x5623f05d7cec]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 27:
> (OSD::ShardedOpWQ::_process(unsigned int,
> ceph::heartbeat_handle_d*)+0x12d0) [0x5623f05179e0]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 28:
> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884)
> [0x5623f0ae44e4]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 29:
> (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5623f0ae7520]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 30: (()+0x76ba) [0x7f526119c6ba]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 31: (clone()+0x6d) [0x7f52602133dd]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 0> 2017-08-02 10:43:32.251841
> 7f524861b700 -1 *** Caught signal (Aborted) **
> Aug 02 10:43:32 osd3 ceph-osd[7413]: in thread 7f524861b700
> thread_name:tp_osd_tp
> Aug 02 10:43:32 osd3 ceph-osd[7413]: ceph version 12.1.2
> (b661348f156f148d764b998b65b90451f096cb27) luminous (rc)
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 1: (()+0xa9a964) [0x5623f0a9c964]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 2: (()+0x11390) [0x7f52611a6390]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 3: (pread64()+0x33) [0x7f52611a5d43]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 4:
> (KernelDevice::direct_read_unaligned(unsigned long, unsigned long,
> char*)+0x81) [0x5623f0a7cfc1]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 5:
> (KernelDevice::read_random(unsigned long, unsigned long, char*,
> bool)+0x4f3) [0x5623f0a7da43]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 6:
> (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long,
> char*)+0x4fa) [0x5623f0a4d9ca]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 7:
> (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
> rocksdb::Slice*, char*) const+0x20) [0x5623f0a77e10]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 8:
> (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
> rocksdb::Slice*, char*) const+0xf8f) [0x5623f0e50acf]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 9:
> (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
> rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle
> const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool,
> rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x5f3)
> [0x5623f0e21383]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 10: (()+0xe0f7c6) [0x5623f0e117c6]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 11:
> (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*,
> rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice,
> rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x2f8)
> [0x5623f0e13928]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 12:
> (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*,
> rocksdb::ReadOptions const&, rocksdb::BlockHandle const&,
> rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x5623f0e13d2c]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 13:
> (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice
> const&)+0x97) [0x5623f0e1c4a7]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 14: (()+0xe4576e) [0x5623f0e4776e]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 15: (()+0xe45836) [0x5623f0e47836]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 16: (()+0xe459b1) [0x5623f0e479b1]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 17:
> (rocksdb::MergingIterator::Next()+0x449) [0x5623f0e2ab09]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 18:
> (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x182)
> [0x5623f0ec7ed2]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 19: (rocksdb::DBIter::Next()+0x1eb)
> [0x5623f0ec8c8b]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 20:
> (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x5623f09dd58a]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 21:
> (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&,
> ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t>
> >*, ghobject_t*)+0x1170) [0x5623f093d250]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 22:
> (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
> ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t,
> std::allocator<ghobject_t> >*, ghobject_t*)+0x25a) [0x5623f093e6ea]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 23:
> (PGBackend::objects_list_range(hobject_t const&, hobject_t const&,
> snapid_t, std::vector<hobject_t, std::allocator<hobject_t> >*,
> std::vector<ghobject_t, std::allocator<ghobject_t> >*)+0x192)
> [0x5623f0700ef2]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 24:
> (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool, unsigned
> int, ThreadPool::TPHandle&)+0x200) [0x5623f05a8b30]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 25:
> (PG::chunky_scrub(ThreadPool::TPHandle&)+0x3ea) [0x5623f05d61ca]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 26: (PG::scrub(unsigned int,
> ThreadPool::TPHandle&)+0x45c) [0x5623f05d7cec]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 27:
> (OSD::ShardedOpWQ::_process(unsigned int,
> ceph::heartbeat_handle_d*)+0x12d0) [0x5623f05179e0]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 28:
> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884)
> [0x5623f0ae44e4]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 29:
> (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5623f0ae7520]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 30: (()+0x76ba) [0x7f526119c6ba]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: 31: (clone()+0x6d) [0x7f52602133dd]
> Aug 02 10:43:32 osd3 ceph-osd[7413]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Aug 02 10:43:32 osd3 systemd[1]: ceph-osd@0.service: Main process exited,
> code=killed, status=6/ABRT
> Aug 02 10:43:32 osd3 systemd[1]: ceph-osd@0.service: Unit entered failed
> state.
> Aug 02 10:43:32 osd3 systemd[1]: ceph-osd@0.service: Failed with result
> 'signal'.
> Aug 02 10:43:52 osd3 systemd[1]: ceph-osd@0.service: Service hold-off
> time over, scheduling restart.
> Aug 02 10:43:52 osd3 systemd[1]: Stopped Ceph object storage daemon osd.0.
> Aug 02 10:43:52 osd3 systemd[1]: Starting Ceph object storage daemon
> osd.0...
> Aug 02 10:43:52 osd3 systemd[1]: Started Ceph object storage daemon osd.0.
> Aug 02 10:43:52 osd3 ceph-osd[8322]: starting osd.0 at - osd_data
> /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
>
> roger@desktop:~$ ceph -s
> cluster:
> id: eea7b78c-b138-40fc-9f3e-3d77afb770f0
> health: HEALTH_WARN
> 1 osds down
> 1 host (1 osds) down
> Degraded data redundancy: 43922/162834 objects degraded
> (26.973%), 300 pgs unclean, 305 pgs degraded
> 114 pgs not deep-scrubbed for 86400
> 155 pgs not scrubbed for 86400
> 10 slow requests are blocked > 32 sec
>
> services:
> mon: 3 daemons, quorum nuc1,nuc2,nuc3
> mgr: nuc3(active), standbys: nuc2, nuc1
> osd: 3 osds: 2 up, 3 in
> rgw: 1 daemon active
>
> data:
> pools: 19 pools, 372 pgs
> objects: 54278 objects, 71724 MB
> usage: 122 GB used, 27819 GB / 27941 GB avail
> pgs: 43922/162834 objects degraded (26.973%)
> 303 active+undersized+degraded
> 67 active+clean
> 2 active+recovery_wait+degraded
>
> roger@desktop:~$ ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 27.28679 root default
> -5 9.09560 host osd1
> 3 hdd 9.09560 osd.3 up 1.00000 1.00000
> -6 9.09560 host osd2
> 4 hdd 9.09560 osd.4 up 1.00000 1.00000
> -2 9.09560 host osd3
> 0 hdd 9.09560 osd.0 down 1.00000 1.00000
>
> roger@desktop:~$ ceph mon versions
> {
> "ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27)
> luminous (rc)": 3
> }
> roger@desktop:~$ ceph osd versions
> {
> "ceph version 12.1.2 (b661348f156f148d764b998b65b90451f096cb27)
> luminous (rc)": 2
> }
>
> roger@osd3:~$ sudo ceph daemon osd.0 status
> {
> "cluster_fsid": "eea7b78c-b138-40fc-9f3e-3d77afb770f0",
> "osd_fsid": "bdb31a03-e381-4bf8-82e3-18916c838308",
> "whoami": 0,
> "state": "waiting_for_healthy",
> "oldest_map": 25389,
> "newest_map": 25938,
> "num_pgs": 372
> }
>
> roger@desktop:~$ ceph df
> GLOBAL:
> SIZE AVAIL RAW USED %RAW USED
> 27941G 27819G 122G 0.44
> POOLS:
> NAME ID USED %USED MAX AVAIL
> OBJECTS
> default.rgw.rgw.gc 70 0 0 8807G
> 0
> default.rgw.buckets.non-ec 83 0 0 8807G
> 43
> default.rgw.control 85 0 0 8807G
> 8
> default.rgw.data.root 86 15601 0 8807G
> 49
> default.rgw.gc 87 0 0 8807G
> 32
> default.rgw.lc 88 0 0 8807G
> 32
> default.rgw.log 89 0 0 8807G
> 144
> default.rgw.users.uid 90 3346 0 8807G
> 14
> default.rgw.users.email 91 100 0 8807G
> 7
> default.rgw.users.keys 92 100 0 8807G
> 7
> default.rgw.buckets.index 93 0 0 8807G
> 39
> default.rgw.intent-log 95 0 0 8807G
> 0
> default.rgw.meta 96 0 0 8807G
> 0
> default.rgw.usage 97 0 0 8807G
> 0
> default.rgw.users.swift 98 39 0 8807G
> 4
> default.rgw.buckets.extra 99 0 0 8807G
> 0
> .rgw.root 100 1681 0 8807G
> 4
> default.rgw.reshard 101 0 0 8807G
> 17
> default.rgw.buckets.data 103 71724M 0.40 17614G
> 53878
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
