[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Laimis Juzeliūnas
Very solid advice here - that’s the beauty of Ceph community. Just adding to what Anthony mentioned: a reweight from 1 to 0.2 (and back) is quite extreme and the cluster won’t like it. We never go above increments/decrements of 0.02-0.04. If you have to go from 1 to 0.98 and then to 0.96 and so

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Anthony D'Atri
> Very solid advice here - that’s the beauty of Ceph community. > > Just adding to what Anthony mentioned: a reweight from 1 to 0.2 (and back) is > quite extreme and the cluster won’t like it. And these days with the balancer, pg-upmap entries to the same effect are a better idea. > From the

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Bruno Gomes Pessanha
Interesting. You're right: # ceph config get osd osd_max_backfills 10 ## ceph-conf --show-config | egrep osd_max_backfills osd_max_backfills = 1 I don't know why that is happening. On Sat, 4 Jan 2025 at 17:13, Laimis Juzeliūnas wrote: > One more question: > What’s the output of 'ceph config get

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Bruno Gomes Pessanha
> > Are you using mclock? I'm not sure. I'll read about it. What happens if you set it through the Ceph UI? There are *global*,* mon*,* mgr*,* osd*,* mds *and* client* values that I changed to 10 (the default is 1): [image: image.png] ...but the output is still: # ceph-conf --show-config | egrep

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Bruno Gomes Pessanha
> > What reweighs have been set for the top OSDs (ceph osd df tree)? > Right now they are all at 1.0. I had to lower them to something close to 0.2 in order to free up space but I changed them back to 1.0. Should I lower them while the backfill is happening? On Sat, 4 Jan 2025 at 17:18, Laimis Juz

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Bruno Gomes Pessanha
> > Do you use the autoscalar or did you trigger a manual PG increment of the > pool? The pool had autoscale enabled until 2 days ago when I thought it was better to change things manually in order to have a more deterministic result. Yes, I wanted to increase from "1" to something like "1024" but

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Anthony D'Atri
>> Do you use the autoscalar or did you trigger a manual PG increment of the >> pool? > > The pool had autoscale enabled until 2 days ago when I thought it was > better to change things manually in order to have a more deterministic > result. Yes, I wanted to increase from "1" to something like "

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

2025-01-05 Thread Anthony D'Atri
>> What reweighs have been set for the top OSDs (ceph osd df tree)? >> > Right now they are all at 1.0. I had to lower them to something close to > 0.2 in order to free up space but I changed them back to 1.0. Should I > lower them while the backfill is happening? Old-style legacy override rewei