Hi Sridhar,
Thanks for the response, I have added the output you requested below, I have
attached the output from the last command in a file as it was rather long. We
did try to set high_recovery_ops but it didn't seem to have any visible effect.
root@gb4-li-cephgw-001 ~ # ceph versions
{
"mon": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
(stable)": 3
},
"mgr": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
(stable)": 3
},
"osd": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
(stable)": 72
},
"mds": {},
"rgw": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
(stable)": 3
},
"overall": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
(stable)": 81
}
}
root@gb4-li-cephgw-001 ~ # ceph config get osd.0 osd_op_queue
mclock_scheduler
root@gb4-li-cephgw-001 ~ # ceph config show osd.0 | grep osd_max_backfills
osd_max_backfills 3
override (mon[3]),default[10]
root@gb4-li-cephgw-001 ~ # ceph config show osd.0 | grep osd_recovery_max_active
osd_recovery_max_active 9
override (mon[9]),default[0]
osd_recovery_max_active_hdd 10
default
osd_recovery_max_active_ssd 20
default
Thanks
Iain
________________________________
From: Sridhar Seshasayee <ssesh...@redhat.com>
Sent: 03 October 2023 09:07
To: Iain Stott <iain.st...@thehutgroup.com>
Cc: ceph-users@ceph.io <ceph-users@ceph.io>; dl-osadmins
<dl-osadm...@thehutgroup.com>
Subject: Re: [ceph-users] Slow recovery and inaccurate recovery figures since
Quincy upgrade
CAUTION: This email originates from outside THG
________________________________
Hello Iain,
Does anyone have any ideas of what could be the issue here or anywhere we can
check what is going on??
You could be hitting the slow backfill/recovery issue with mclock_scheduler.
Could you please provide the output of the following commands?
1. ceph versions
2. ceph config get osd.<id> osd_op_queue
3. ceph config show osd.<id> | grep osd_max_backfills
4. ceph config show osd.<id> | grep osd_recovery_max_active
5. ceph config show-with-defaults osd.<id> | grep osd_mclock where 'id' can be
any valid osd id
With the mclock_scheduler enabled and with 17.2.5, it is not possible to
override
recovery settings like 'osd_max_backfills' and other recovery related config
options.
To improve the recovery rate, you can temporarily switch the mClock profile to
'high_recovery_ops'
on all the OSDs by issuing:
ceph config set osd osd_mclock_profile high_recovery_ops
During recovery with this profile, you may notice a dip in the client ops
performance which is expected.
Once the recovery is done, you can switch the mClock profile back to the
default 'high_client_ops' profile.
Please note that the upcoming Quincy release will address the slow backfill
issues along with other
usability improvements.
-Sridhar
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
