[ceph-users] Ceph v.15.2.15 (Octopus, stable) - OSD_SCRUB_ERRORS: 6 scrub errors

Hello, My Ceph cluster with 3 nodes is showing a HEALTH_ERR, with the following errors: * OSD_SCRUB_ERRORS: 6 scrub errors * PG_DAMAGED: Possible data damage: 6 pgs inconsistent * CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) * MON_CLOCK_SKEW: clock skew detected on mon.ceph3 * MON_DO

[ceph-users] Re: Using CephFS in High Performance (and Throughput) Compute Use Cases

Hi Manuel! I'm the one that submitted the the io500 results for Red Hat. Ok, so a couple of things.  First be aware that vendors are not required to use any form of replication whatsoever for the IO500 test submissions.  Our results are thus using 1x replication. :) But!  2x should only hur

[ceph-users] Cephadm + OpenStack Keystone Authentication

Hello everyone, I'm currently having a problem to use Cephadm and integrate the RadosGW and Object Storage into OpenStack. If I try to use Object Storage via Swift in OpenStack it does not work. While trying in Horizon, I simply get logged out of the admin user with the error message: "Unauth

[ceph-users] Re: Stop Rebalancing

One more thing, could you please also share the `ceph osd pool autoscale-status` ? On Tue, Apr 12, 2022 at 9:50 PM Ray Cunningham wrote: > > Thank you Dan! I will definitely disable autoscaler on the rest of our pools. > I can't get the PG numbers today, but I will try to get them tomorrow. We

[ceph-users] Re: Stop Rebalancing

Perfect timing, I was just about to reply. We have disabled autoscaler on all pools now. Unfortunately, I can't just copy and paste from this system... `ceph osd pool ls detail` only 2 pools have any difference. pool1: pgnum 940, pgnum target 256, pgpnum 926 pgpnum target 256 pool7: pgnum 2

[ceph-users] Re: Ceph Developer Summit - Reef

Hi everyone, The summit continues in five minutes with topics on RGW: https://bluejeans.com/908675367/ https://pad.ceph.com/p/cds-reef On Tue, Apr 12, 2022 at 8:11 AM Mike Perez wrote: > > Hi everyone, > > The Ceph Developer Summit for Reef is now starting on discussions on > the Orchestrator >

[ceph-users] Re: Stop Rebalancing

All pools have gone backfillfull. Thank you, Ray Cunningham Systems Engineering and Services Manager keepertechnology (571) 223-7242 From: Ray Cunningham Sent: Wednesday, April 13, 2022 10:15:56 AM To: Dan van der Ster Cc: ceph-

[ceph-users] Re: Stop Rebalancing

No repair IO and misplaced objects increasing with norebalance and nobackfill set. Thank you, Ray From: Ray Cunningham Sent: Wednesday, April 13, 2022 10:38:29 AM To: Dan van der Ster Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Stop Rebalancing All poo

[ceph-users] Re: Stop Rebalancing

Hi, Thanks. norebalance/nobackfill are useful to pause ongoing backfilling, but aren't the best option now to get the PGs to go active+clean and let the mon db come back under control. Unset those before continuing. I think you need to set the pg_num for pool1 to something close to but less than

[ceph-users] Call for Submissions IO500 ISC 2022 list

Stabilization Period: Monday, April 4th - Friday, April 15th Submission Deadline: Friday, May 13th, 2022 AoE The IO500 is now accepting and encouraging submissions for the upcoming 10th semi-annual IO500 list, in conjunction with ISC-HPC'22. Once again, we are also accepting submissions to th

[ceph-users] Re: Stop Rebalancing

Thank you so much, Dan! Can you confirm for me that for pool7, which has 2048/2048 for pg_num and 883/2048 for pgp_num, we should change pg_num or pgp_num? And can they be different for a single pool, or does pg_num and pgp_num have to always be the same? IF we just set pgp_num to 890 we wil

[ceph-users] Re: Stop Rebalancing

I would set the pg_num, not pgp_num. In older versions of ceph you could manipulate these things separately, but in pacific I'm not confident about what setting pgp_num directly will do in this exact scenario. To understand, the difference between these two depends on if you're splitting or mergin

[ceph-users] Re: Stop Rebalancing

On Wed, Apr 13, 2022 at 10:01 AM Dan van der Ster wrote: > > I would set the pg_num, not pgp_num. In older versions of ceph you could > manipulate these things separately, but in pacific I'm not confident about > what setting pgp_num directly will do in this exact scenario. > > To understand, the

[ceph-users] Re: Stop Rebalancing

Ok, so in our situation with high pg_num and a low pgp_num, is there any way we can make it stop backfilling temporarily? The system is already operating with different pg and pgp numbers, so I'm thinking it won't kill the cluster if we just set the pgp_num and make it stop splitting for the mom

[ceph-users] Re: Stop Rebalancing

On Wed, Apr 13, 2022 at 7:07 PM Gregory Farnum wrote: > > On Wed, Apr 13, 2022 at 10:01 AM Dan van der Ster wrote: > > > > I would set the pg_num, not pgp_num. In older versions of ceph you could > > manipulate these things separately, but in pacific I'm not confident about > > what setting pgp_n

[ceph-users] Re: Stop Rebalancing

> In any case, isn't this still the best approach to make all PGs go > active+clean ASAP in this scenario? > > 1. turn off the autoscaler (for those pools, or fully) > 2. for any pool with pg_num_target or pgp_num_target values, get the > current pgp_num X and use it to `ceph osd pool set pg_nu

[ceph-users] Re: Stop Rebalancing

We've done that, I'll update with what happens overnight. Thanks everyone! Thank you, Ray From: Anthony D'Atri Sent: Wednesday, April 13, 2022 4:49 PM To: Ceph Users Subject: [ceph-users] Re: Stop Rebalancing > In any case, isn't this still the best approac

[ceph-users] Re: Stop Rebalancing

For the moment, Dan's workaround sounds good to me, but I'd like to understand how we got here, in terms of the decisions that were made by the autoscaler. We have a config option called "target_max_misplaced_ratio" (default value is 0.05), which is supposed to limit the number of misplaced objects

[ceph-users] Re: quincy v17.2.0 QE Validation status

On Mon, Apr 11, 2022 at 7:33 PM Venky Shankar wrote: > > On Fri, Apr 8, 2022 at 3:32 PM Venky Shankar wrote: > > > > On Tue, Apr 5, 2022 at 7:44 AM Venky Shankar wrote: > > > > > > Hey Josh, > > > > > > On Tue, Apr 5, 2022 at 4:34 AM Josh Durgin wrote: > > > > > > > > Hi Venky and Ernesto, how