Hi,
We are observing a new strange behaviour on our production cluster : we
increased the number of PG (from 256 to 2048) in a (EC) pool after a
warning that there was a very high number of objects per pool (the pool
has 52M objects).
Background: this happens in the cluster that had a strange problem last
week, discussed in the thread "Production cluster in bad shape after
several OSD crashes". The PG increase was done after the cluster
returned to a normal state.
The increase of the number of PG resulted in 20% misplaced objects and
~160 PG remapped (over 256). As there is no much user activity on this
cluster (except on this pool) these days, we decided to set the mclock
profile to high_recovery_ops. We also disabled autoscaler on this pool
(it was enabled and it is not clear why we add the warning with
autoscaler enabled). The pool was created with --bulk.
The remapping went steadily during 2-3 days (as much as we can tell) but
when reaching the end (between 0.5% and 1% of misplaced objects, ~10 PG
remapped) it readds remapped PG (that can be seen with 'ceph pg
dump_stuck'), all belonging to the pool affected by the increase. This
already happened 3-4 times and it is very unclear why. There was no
specific problem reported on the cluster that may explain this (no OSD
down). I was wondering if the balancer may be responsible for this but I
don't have the feeling it is the case: first the balancer doesn't report
about doing something (but I may miss the history), second the balancer
would probably affect PG from different pools. (there are 50 ppols in
the cluster). There are 2 warnings that may or may not be related:
- All mon have their data size > 15 GB (mon_data_size_warn). Currently
16 GB. It happened progressively during the remapping on all 3 mon, I
guess it is due to the operation in progress and is harmless. Do you
confirm?
- Since yesterday we have 4 PG that have not been deep scrubbed in time,
belonging to different pools. Again, I tend to correlate this to the
remapping in progress putting too much load (or other constraints), as
there are a lot of deep scrubs per day. The current age distribution of
deep scrubs is:
4 "2025-03-19
21 "2025-03-20
46 "2025-03-21
35 "2025-03-22
81 "2025-03-23
597 "2025-03-24
1446 "2025-03-25
2234 "2025-03-26
2256 "2025-03-27
1625 "2025-03-28
1980 "2025-03-29
2993 "2025-03-30
3871 "2025-03-31
1113 "2025-04-01
Should we worry about the situation? If yes, what would you advise to
look at or do? To clear the problem last week, we had to restart all OSD
but we didn't restart mon. Do they play a role in deciding the remapping
plan? Is restarting them something that may help?
As usual, thanks in advance for any help/hint.
Best regards,
Michel
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
