Hi Bukhard,
Thanks for your answer. Your explanation seems to match well our
observations, in particular the fact that new misplaced objects are
added when we fall under something like 0.5% of misplaced objects. What
is not clear for me anyway is that 'ceph osd pool ls detail' for the
pool modified is not reporting the new pg_num target (2048) but the old
one (256):
pool 62 'ias-z1.rgw.buckets.data' erasure profile k9_m6_host size 15
min_size 10 crush_rule 3 object_hash rjenkins pg_num 323 pgp_num 307
pg_num_target 256 pgp_num_target 256 autoscale_mode off last_change
439681 lfor 0/439680/439678 flags hashpspool,bulk max_bytes
200000000000000 stripe_width 36864 application rgw
- Is this caused by the fact that autoscaler was still on when I
increased the number of PG and that I disabled it on the pool ~12h after
entering the command to extend it?
- Or was it a mistake of mine to enter only extend the pg_num and not
the pgp_num. According to the doc that I just read again, both should be
extended at the same time or it not causing the expected result? If it
is the case, should I just reenter the command to extend pg_num and
pgp_num? (and wait for the resulting remapping!).
Best regards,
Michel
Le 01/04/2025 à 09:32, Burkhard Linke a écrit :
Hi,
On 4/1/25 09:06, Michel Jouvin wrote:
Hi,
We are observing a new strange behaviour on our production cluster :
we increased the number of PG (from 256 to 2048) in a (EC) pool after
a warning that there was a very high number of objects per pool (the
pool has 52M objects).
Background: this happens in the cluster that had a strange problem
last week, discussed in the thread "Production cluster in bad shape
after several OSD crashes". The PG increase was done after the
cluster returned to a normal state.
The increase of the number of PG resulted in 20% misplaced objects
and ~160 PG remapped (over 256). As there is no much user activity on
this cluster (except on this pool) these days, we decided to set the
mclock profile to high_recovery_ops. We also disabled autoscaler on
this pool (it was enabled and it is not clear why we add the warning
with autoscaler enabled). The pool was created with --bulk.
The remapping went steadily during 2-3 days (as much as we can tell)
but when reaching the end (between 0.5% and 1% of misplaced objects,
~10 PG remapped) it readds remapped PG (that can be seen with 'ceph
pg dump_stuck'), all belonging to the pool affected by the increase.
This already happened 3-4 times and it is very unclear why. There was
no specific problem reported on the cluster that may explain this (no
OSD down). I was wondering if the balancer may be responsible for
this but I don't have the feeling it is the case: first the balancer
doesn't report about doing something (but I may miss the history),
second the balancer would probably affect PG from different pools.
(there are 50 ppols in the cluster). There are 2 warnings that may or
may not be related:
*snipsnap*
you are probably seeing the loadbalancer in action. If you increase
the number of PGs, this change is performed gradually. You can check
this by having a look at the output of 'ceph osd pool ls detail'. It
will print the current number of PGs _and_ PGPs, and the target number
of both values. If you increase the number of PG, existing PGs have to
be splitted and a part of the data has to be transferred to the new
PG. The loadbalancer will take care for this and start the next PG
split if the number of misplaced objects falls under a certain
threshold. The default value is 0.5% is I remember correctly.
- All mon have their data size > 15 GB (mon_data_size_warn).
Currently 16 GB. It happened progressively during the remapping on
all 3 mon, I guess it is due to the operation in progress and is
harmless. Do you confirm?
The mons have to save historic osd and pg maps as long as the cluster
is not healthy. During large data movement operations this might pile
up to a significant size. You should monitor the available space and
ensure that the mons are not running out of disk space. I'm not sure
whether manual intervention like a manual compaction will help in this
case
- Since yesterday we have 4 PG that have not been deep scrubbed in
time, belonging to different pools. Again, I tend to correlate this
to the remapping in progress putting too much load (or other
constraints), as there are a lot of deep scrubs per day. The current
age distribution of deep scrubs is:
4 "2025-03-19
21 "2025-03-20
46 "2025-03-21
35 "2025-03-22
81 "2025-03-23
597 "2025-03-24
1446 "2025-03-25
2234 "2025-03-26
2256 "2025-03-27
1625 "2025-03-28
1980 "2025-03-29
2993 "2025-03-30
3871 "2025-03-31
1113 "2025-04-01
Should we worry about the situation? If yes, what would you advise to
look at or do? To clear the problem last week, we had to restart all
OSD but we didn't restart mon. Do they play a role in deciding the
remapping plan? Is restarting them something that may help?
Remapping PGs has a higher priority than scrubbing. So as long as the
current pool extension is not finished, only idle OSDs will scrub
their PGs. This is expected, and the cluster will take care for the
missing scrub runs after it is healthy again.
Best regards,
Burkhard Linke
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
