Hi
19.2.2 with WPQ, osd-max-backfills and osd-recovery-max-active are set to 1.
In order to get our PG sizes better aligned we doubled the number of PGs
on the pool with the largest PG size. The pool is HDD with DB/WAL on
SATA SSD and HDD sizes between 2TB and 20TB and PG size was ~140GB
before the doubling.
All went well but looking at this:
"
cluster:
id:
health: HEALTH_OK
services:
mon: 5 daemons, quorum
ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 2d)
mgr: ceph-flash2.utlhuz(active, since 4d), standbys:
ceph-flash3.ciudre, ceph-flash1.erhakb
mds: 1/1 daemons up, 2 standby
osd: 576 osds: 576 up (since 2h), 576 in (since 3d); 8767 remapped pgs
data:
volumes: 1/1 healthy
pools: 18 pools, 25249 pgs
objects: 683.85M objects, 1.6 PiB
usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail
pgs: 842769842/3951610673 objects misplaced (21.327%)
16481 active+clean
8762 active+remapped+backfill_wait
6 active+remapped+backfilling
io:
client: 374 MiB/s rd, 14 MiB/s wr, 2.86k op/s rd, 410 op/s wr
recovery: 153 MiB/s, 38 objects/s
"
The balancer was running and seemingly making very small changes:
"
[root@lazy ~]# ceph balancer status
{
"active": true,
"last_optimize_duration": "0:00:01.012679",
"last_optimize_started": "Mon Apr 28 10:01:24 2025",
"mode": "upmap",
"no_optimization_needed": true,
"optimize_result": "Optimization plan created successfully",
"plans": []
}
"
I tried the pgremapper and that gave a nice throughput in the beginning
but now it seems to be about back to scratch:
"
cluster:
id:
health: HEALTH_OK
services:
mon: 5 daemons, quorum
ceph-flash1,ceph-flash2,ceph-flash3,ceph-e3s2,ceph-e3s3 (age 7h)
mgr: ceph-flash2.utlhuz(active, since 5d), standbys:
ceph-flash3.ciudre, ceph-flash1.erhakb
mds: 1/1 daemons up, 2 standby
osd: 577 osds: 577 up (since 5h), 577 in (since 31h); 7392 remapped pgs
data:
volumes: 1/1 healthy
pools: 18 pools, 25249 pgs
objects: 683.96M objects, 1.6 PiB
usage: 2.7 PiB used, 1.9 PiB / 4.5 PiB avail
pgs: 459889823/3952312686 objects misplaced (11.636%)
17857 active+clean
7384 active+remapped+backfill_wait
8 active+remapped+backfilling
io:
client: 70 MiB/s rd, 15 MiB/s wr, 689 op/s rd, 378 op/s wr
recovery: 294 MiB/s, 75 objects/s
"
Balancer is still running and making small changes.
This is going to take a while, any tips on how to escape the apparent
bottleneck?
Is having many PGs misplaced actually counter productive, so it would be
better to throttle via target_max_misplaced_ratio? I was thinking it was
better to let the balancer balance all it could, as that would make all
the moves available and decrease the risk of bottlenecking.
Thanks.
Mvh.
Torkil
--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
