Hi Everyone!
I have a 16 node, 640 OSD (5 to 1 SSD) bluestore cluster which is mainly used
for RGW services. It has its own backend cluster network for IO separate from
the customer network.
Whenever we add or remove an OSD the rebalance or repair IO starts off very
fast 4GB/s+ but it will con
I can do that for adding new drives, but what about when removing failed disks?
There’s not one OSD to run an iostat on? The performance profile is the same
for adds and removes.
Thank you,
Ray
From: David Orman
Sent: Thursday, January 6, 2022 9:18 AM
To: Ray Cunningham
Cc: ceph-users
stions for increasing scrub throughput?
Thank you,
Ray Cunningham
Systems Engineering and Services Manager
keepertechnology<http://www.keepertech.com/>
(571) 223-7242
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email
n get answers for specific
questions.
Off the top of my head, we have set:
osd_max_scrubs 20
osd_scrub_auto_repair true
osd _scrub_load_threashold 0.6
We do not limit srub hours.
Thank you,
Ray
-Original Message-
From: norman.kern
Sent: Wednesday, March 9, 2022 7:28 PM
To: Ray Cunni
We have that set to 20 at the moment.
Thank you,
Ray Cunningham
Systems Engineering and Services Manager
keepertechnology<http://www.keepertech.com/>
(571) 223-7242
From: Szabo, Istvan (Agoda)
Sent: Wednesday, March 9, 2022 7:35 PM
To: Ray Cunningham
Cc: ceph-users@ceph.io
Subject: Re:
load is? Is that a ceph command, or an OS command?
Thank you,
Ray
-Original Message-
From: Ray Cunningham
Sent: Thursday, March 10, 2022 7:59 AM
To: norman.kern
Cc: ceph-users@ceph.io
Subject: RE: [ceph-users] Scrubbing
We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for
Well that was incorrect. Someone changed it back to 1. I have now set our max
scrubs to 2. We’ll see if that makes a difference.
Thank you,
Ray
From: Ray Cunningham
Sent: Thursday, March 10, 2022 8:00 AM
To: Szabo, Istvan (Agoda)
Cc: ceph-users@ceph.io
Subject: RE: [ceph-users] Scrubbing
We
;ve been bottlenecked by those?
Thank you,
Ray
-Original Message-
From: norman.kern
Sent: Thursday, March 10, 2022 9:27
To: Ray Cunningham
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Re: Scrubbing
Ray,
You can use node-exporter+prom+grafana to collect the load of CPUs statistics.
mance impact.
dp
On 3/11/22 17:32, Ray Cunningham wrote:
> That's what I thought. We looked at the cluster storage nodes and found them
> all to be less than .2 normalized maximum load.
>
> Our 'normal' BW for client IO according to ceph -s is around 60MB/s-100MB/s.
&g
Hi Everyone,
We just upgraded our 640 OSD cluster to Ceph 16.2.7 and the resulting
rebalancing of misplaced objects is overwhelming the cluster and impacting MON
DB compaction, deep scrub repairs and us upgrading legacy bluestore OSDs. We
have to pause the rebalancing if misplaced objects or we
including the term premerge so you
are probably right about this being caused by a PG merge.
Thank you,
Ray
-Original Message-
From: Matt Vandermeulen
Sent: Tuesday, April 12, 2022 2:39 PM
To: Ray Cunningham
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Stop Rebalancing
It soun
il 12, 2022 2:46 PM
To: Ray Cunningham
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Stop Rebalancing
Hi Ray,
Disabling the autoscaler on all pools is probably a good idea. At least until
https://tracker.ceph.com/issues/53729 is fixed. (You are likely not susceptible
to that -- but better safe
: Dan van der Ster
Sent: Wednesday, April 13, 2022 10:00 AM
To: Ray Cunningham
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Stop Rebalancing
One more thing, could you please also share the `ceph osd pool
autoscale-status` ?
On Tue, Apr 12, 2022 at 9:50 PM Ray Cunningham
wrote:
>
> Th
All pools have gone backfillfull.
Thank you,
Ray Cunningham
Systems Engineering and Services Manager
keepertechnology<http://www.keepertech.com/>
(571) 223-7242
From: Ray Cunningham
Sent: Wednesday, April 13, 2022 10:15:56 AM
To: Dan van der Ster
Cc
No repair IO and misplaced objects increasing with norebalance and nobackfill
set.
Thank you,
Ray
From: Ray Cunningham
Sent: Wednesday, April 13, 2022 10:38:29 AM
To: Dan van der Ster
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Stop Rebalancing
All
#x27;ll have to mute a warning in the meantime -- check the docs after the
warning appears).
Cheers, dan
On Wed, Apr 13, 2022 at 5:16 PM Ray Cunningham
wrote:
>
> Perfect timing, I was just about to reply. We have disabled autoscaler on all
> pools now.
>
> Unfortunately, I can&#
ng for the moment..
Thank you,
Ray
-Original Message-
From: Gregory Farnum
Sent: Wednesday, April 13, 2022 12:07 PM
To: Dan van der Ster
Cc: Ray Cunningham ; Ceph Users
Subject: Re: [ceph-users] Re: Stop Rebalancing
On Wed, Apr 13, 2022 at 10:01 AM Dan van der Ster wrote:
>
>
We've done that, I'll update with what happens overnight. Thanks everyone!
Thank you,
Ray
From: Anthony D'Atri
Sent: Wednesday, April 13, 2022 4:49 PM
To: Ceph Users
Subject: [ceph-users] Re: Stop Rebalancing
> In any case, isn't this still the best approac
18 matches
Mail list logo
