On 7/7/25 18:34, mhnx wrote:
Hello!
Few years ago I build a "dc-a:12 + dc-b:12 = 24" node ceph cluster
with Nautilus v14.2.16
A year ago the cluster upgraded to Octopus and it was running fine.
Recently I added 4+4=8 new nodes with identical hardware and SSD drives.
When I created OSD's with Octopus, The cluster usage increased from %50 to %78!!
What does a "ceph osd df tree" gives you?
The weird problem is, the new OSD's become nearfull and hold more size
even if they have the same or less amount of PG's.
I had to reweight new OSD's to 0.9 to make them equal size usage..
I increased the PG count 8192 to 16384 and ran balancer, it became
worse and I have %84 usage now!
Remember that Ceph is limited by the fullest OSD in the cluster.
Do you have old clients? If not, try to get rid of reweight and start
using upmap. It is way more efficient in getting a cluster well
balanced. I would recommend using this balance script:
https://github.com/TheJJ/ceph-balancer
Maybe first reset all the reweigths (first do: ceph osd set nobackfill).
Then run this script:
https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remapped.py
And after that run the ceph-balancer script. That should help
tremendously if the cluster is imbalanced.
I guess OSD or PG code changed between nautilus <-> octopus and it
generates this problem.
What version of Octopus are you running?
Can anyone help me with experience or knowledge about this?
What should I do?
My solution idea:
I'm thinking of destroy and re-create old OSD's as a solution but I
need to re-create 144x3.8TB Sas SSD OSD's and it means 4-5 days of
maintenance.
Also I have 2 osd per drive because it was recommended at Nautilus
times. How about this? Should I keep the config or should I use 1 osd
per 3.8TB SAS SSD ? What is the recommendation for Octopus and Quincy?
I would recommend upgrading to newer, supported versions, maybe go to
Pacific and then Reef. Modern versions of Ceph do not gain from
deploying multiple OSDs per drive. What Ceph services are you running
(MDS, RGW, RBD)?
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
