[ceph-users] hdd pg's migrating when converting ssd class osd's

[Marc Roos]

I have been converting ssd's osd's to dmcrypt, and I have noticed that 
pg's of pools are migrated that should be (and are?) on hdd class. 

On a healthy ok cluster I am getting, when I set the crush reweight to 
0.0 of a ssd osd this:

17.35 10415  00  9907   0  
36001743890   0  0 3045 3045 
active+remapped+backfilling 2020-09-27 12:55:49.093054  83758'20725398 
83758:100379720  [8,14,23]  8  [3,14,23]  3  
83636'20718129 2020-09-27 00:58:07.098096  83300'20689151 2020-09-24 
21:42:07.385360 0

However osds 3,14,23,8 are all hdd osd's

Since this is a cluster from Kraken/Luminous, I am not sure if the 
device class of the replicated_ruleset[1] was set when the pool 17 was 
created. 
Weird thing is that all pg's of this pool seem to be on hdd osd[2]

Q. How can I display the definition of 'crush_rule 0' at the time of the 
pool creation? (To be sure it had already this device class hdd 
configured)



[1]
[@~]# ceph osd pool ls detail | grep 'pool 17'
pool 17 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 83712 
flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd


[@~]# ceph osd crush rule dump replicated_ruleset
{
"rule_id": 0,
"rule_name": "replicated_ruleset",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -10,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}

[2]
[@~]# for osd in `ceph pg dump pgs| grep '^17' | awk '{print $17" "$19}' 
| grep -oE '[0-9]{1,2}'| sort -u -n`; do ceph osd crush get-device-class 
osd.$osd ; done | sort -u
dumped pgs
hdd
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus

[Igor Fedotov]

On 9/25/2020 6:07 PM, sa...@planethoster.info wrote:

Hi Igor,

The only thing abnormal about this osdstore is that it was created by 
Mimic 13.2.8 and I can see that the OSDs size of this osdstore are not 
the same as the others in the cluster (while they should be exactly 
the same size).


Can it be https://tracker.ceph.com/issues/39151 ?


hmm, may be... Did you change H/W at some point for this OSD's node as 
it happened in the ticket?


And it's still unclear to me if the issue is reproducible for you.

Could you please also run fsck (at first) and then repair for this OSD 
and collect log(s).



Thanks,

Igor




Thanks!
Saber
CTO @PlanetHoster

On Sep 25, 2020, at 5:46 AM, Igor Fedotov > wrote:


Hi Saber,

I don't think this is related. New assertion happens along the write 
path while the original one occurred on allocator shutdown.



Unfortunately there are not much information to  troubleshoot this... 
Are you able to reproduce the case?



Thanks,

Igor

On 9/25/2020 4:21 AM, sa...@planethoster.info wrote:

Hi Igor,

We had an osd crash a week after running Nautilus. I have attached 
the logs, is it related to the same bug?





Thanks,
Saber
CTO @PlanetHoster

On Sep 14, 2020, at 10:22 AM, Igor Fedotov > wrote:


Thanks!

Now got the root cause. The fix is on its way...

Meanwhile you might want to try to workaround the issue via setting 
"bluestore_hybrid_alloc_mem_cap" to 0 or using different allocator, 
e.g. avl for bluestore_allocator (and optionally for 
bluefs_allocator too).



Hope this helps,

Igor.



On 9/14/2020 5:02 PM, Jean-Philippe Méthot wrote:

Alright, here’s the full log file.





Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
4414-4416 Louis B Mayer
Laval, QC, H7P 0G1, Canada
TEL : +1.514.802.1644 - Poste : 2644
FAX : +1.514.612.0678
CA/US : 1.855.774.4678
FR : 01 76 60 41 43
UK : 0808 189 0423






Le 14 sept. 2020 à 06:49, Igor Fedotov > a écrit :


Well, I can see duplicate admin socket command 
registration/de-registration (and the second de-registration 
asserts) but don't understand how this could happen.


Would you share the full log, please?


Thanks,

Igor

On 9/11/2020 7:26 PM, Jean-Philippe Méthot wrote:

Here’s the out file, as requested.




Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
4414-4416 Louis B Mayer
Laval, QC, H7P 0G1, Canada
TEL : +1.514.802.1644 - Poste : 2644
FAX : +1.514.612.0678
CA/US : 1.855.774.4678
FR : 01 76 60 41 43
UK : 0808 189 0423






Le 11 sept. 2020 à 10:38, Igor Fedotov > a écrit :


Could you please run:

CEPH_ARGS="--log-file log --debug-asok 5" ceph-bluestore-tool 
repair --path <...> ; cat log | grep asok > out


and share 'out' file.


Thanks,

Igor

On 9/11/2020 5:15 PM, Jean-Philippe Méthot wrote:

Hi,

We’re upgrading our cluster OSD node per OSD node to Nautilus 
from Mimic. From some release notes, it was recommended to run 
the following command to fix stats after an upgrade :


ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0

However, running that command gives us the following error 
message:


/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc 
: In
 function 'virtual Allocator::SocketHook::~SocketHook()' 
thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc 
: 53

: FAILED ceph_assert(r == 0)
 ceph version 14.2.11 
(f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, 
char const*)+0x14a) [0x7f1a5a823025]

 2: (()+0x25c1ed) [0x7f1a5a8231ed]
 3: (()+0x3c7a4f) [0x55b33537ca4f]
 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517]
 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082]
 6: (BlueStore::_close_db_and_around(bool)+0x2f8) 
[0x55b335274528]
 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) 
[0x55b3352749a1]

 8: (main()+0x10b3) [0x55b335187493]
 9: (__libc_start_main()+0xf5) [0x7f1a574aa555]
 10: (()+0x1f9b5f) [0x55b3351aeb5f]
2020-09-10 14:40:25.873 7f1a6467eec0 -1 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc 
: In function 'virtual 
Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 
time 2020-09-10 14:40:25.872353
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86

[ceph-users] Re: hdd pg's migrating when converting ssd class osd's

[Stefan Kooman]

On 2020-09-27 14:05, Marc Roos wrote:

> Q. How can I display the definition of 'crush_rule 0' at the time of the 
> pool creation? (To be sure it had already this device class hdd 
> configured)

Interesting question. I don't think that information is stored in Ceph
somewhere. But it would be very useful. Similar to what zfs is doing
with "zpool history". I find that very helpful to look back and see what
has and hasn't been done in the past, i.e. ceph pool $pool history

/me heads off to issue tracker to file a feature request ...

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io