[ceph-users] Re: 3 node CEPH PVE hyper-converged cluster serious fragmentation and performance loss in matter of days.

Hi Sasa, jsut a few thoughts/questions on your issue in attempt to understand what's happening. First of all I'd like to clarify what exact command are you using to assess the fragmentation. There are two options: "bluestore allocator score" and "bluestore allocator fragmentation" Both are

[ceph-users] Scrubs stalled on Pacific

Hello Ceph team. I've recently (3 weeks ago) upgraded a Ceph cluster from Nautilus to Pacific (16.2.7), and have encountered a strange issue. Scrubs, either deep or not, are scheduled and are shown in the cluster status, but there is no disk IO and they never finish. At the moment all of my PGs h

[ceph-users] OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats

Hi, I’m in the process of upgrading all our ceph servers from 14.2.9 to 16.2.7. Two of three monitors are on 16.2.6 and one is 16.2.7. I will update them soon. Before updating to 16.2.6/7 I set the “bluestore_fsck_quick_fix_on_mount false” flag and I already upgraded more than the half of my O

[ceph-users] Re: OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats

> > I’m in the process of upgrading all our ceph servers from 14.2.9 to > 16.2.7. Why not first to 14.2.22? I think the standard is to upgrade from the newest version, not? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an ema

[ceph-users] Replace HDD with cephadm

Hello I have a Ceph Pacific cluster managed by cephadm. The nodes have six HDD:s and one NVME that is shared between the six HDD:s. The OSD spec file looks like this: service_type: osdservice_id: osd_spec_defaultplacement:  host_pattern: '*'data_devices:  rotational: 1db_devices:  rotational: 0  si

[ceph-users] Re: Scrubbing

We have 16 Storage Servers each with 16TB HDDs and 2TB SSDs for DB/WAL, so we are using bluestore. The system is running Nautilus 14.2.19 at the moment, with an upgrade scheduled this month. I can't give you a complete ceph config dump as this is an offline customer system, but I can get answer

[ceph-users] Re: Scrubbing

We have that set to 20 at the moment. Thank you, Ray Cunningham Systems Engineering and Services Manager keepertechnology (571) 223-7242 From: Szabo, Istvan (Agoda) Sent: Wednesday, March 9, 2022 7:35 PM To: Ray Cunningham Cc: ceph-users@ceph.io Subject: Re: [ceph-

[ceph-users] ceph -s hang at Futex: futex_wait_setbit_private:futex_clock_realtime

Hi Ceph experts Recently, I try to fix a ceph cluster with all monitor crashed. I created a new monitor with cephx disabled. Then I use the ceph-objectstore-tool to dump cluster map from all ceph osds. i move the the .sst files to a new folder then use "ceph-monstore-tool /tmp/monstore/ rebuild

[ceph-users] Re: Scrubbing

From: osd_scrub_load_threshold The normalized maximum load. Ceph will not scrub when the system load (as defined by getloadavg() / number of online CPUs) is higher than this number. Default is 0.5. Does anyone know how I can run getloadavg() / number of online CPUs so I can see what our load

[ceph-users] Re: 3 node CEPH PVE hyper-converged cluster serious fragmentation and performance loss in matter of days.

> First of all I'd like to clarify what exact command are you using to > assess the fragmentation. There are two options: "bluestore allocator > score" and "bluestore allocator fragmentation" I am using this one : "ceph daemon osd.$i bluestore allocator score block" > Both are not very accurate th

[ceph-users] Procedure for migrating wal.db to ssd

Hi Everyone, I am running a containerized pacific cluster 15.2.15 with 80 spinning disks and 20 SSD. Currently the SSDs are being used as a cach tier and holds the metadata pool for cephfs. I think we could make better use of the SSDs by moving block.wal and block.db to the SSDs and I have a fe

[ceph-users] Re: Scrubbing

Well that was incorrect. Someone changed it back to 1. I have now set our max scrubs to 2. We’ll see if that makes a difference. Thank you, Ray From: Ray Cunningham Sent: Thursday, March 10, 2022 8:00 AM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Scrubbing We ha

[ceph-users] Election deadlock after network split in stretch cluster

Hi, We have a test cluster for evaluating stretch mode and we are running into an issue where the monitors fail to elect a leader after a network split, even though all monitors are online and can reach each other. I saw a presentation by Gregory Farnum from FOSDEM 2020 about stretch clusters and

[ceph-users] Re: Election deadlock after network split in stretch cluster

On Thu, Mar 10, 2022 at 06:33:10PM +0100, Florian Pritz wrote: > We have a test cluster for evaluating stretch mode and we are running > into an issue where the monitors fail to elect a leader after a network > split, even though all monitors are online and can reach each other. Oh, totally forg

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

Hi Igor! I hope I've cracked the checkpot now. I have logs with osd debug level 20 for bluefs, bdev, and bluestore. The log files ceph-osd.4.log shows 2 consecutive startups of the osd.4 where the second startup results in: ``` rocksdb: Corruption: Bad table magic number: expected 98635183903770

[ceph-users] Re: OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats

Hi, After Nautilus there were two omap usage stats upgrades: Octopus (v15) fsck (on by default) enables per-pool omap usage stats. Pacific (v16) fsck (off by default) enables per-pg omap usage stats. (fsck is off by default in pacific because it takes quite some time to update the on-disk metadata

[ceph-users] Re: Scrubbing

Ray, You can use node-exporter+prom+grafana  to collect the load of CPUs statistics. You can use uptime command to get the current statistics. On 3/10/22 10:51 PM, Ray Cunningham wrote: From: osd_scrub_load_threshold The normalized maximum load. Ceph will not scrub when the system load (as d

[ceph-users] Re: Scrubbing

Ray, Do you known the IOPS/BW of the cluster?  The 16TB HDD is more suitable for cold data, If the clients' bw/iops is too big, you can never  finish the scrub. And if you adjust the priority, it will have a great impact to the clients. On 3/10/22 9:59 PM, Ray Cunningham wrote: We have 16 Stor

[ceph-users] Re: mclock and background best effort

Hello Luis, Background best effort includes background operations such as scrubbing, PG deletion, and snapshot trimming. Hope that answers your question! Regards, Aishwarya Mathuria ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send a

[ceph-users] Re: OSD storage not balancing properly when crush map uses multiple device classes

Hi, i think i have a similar problem with my Octopus cluster. $ ceph osd df | grep ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 31 ssd 0.36400 1.0 373 GiB 289 GiB 258 GiB 91 MiB 31 GiB 84 GiB 77.46 1.20 110 up ===> not rebalanced, is it normal? 46 s

[ceph-users] rbd namespace create - operation not supported

Hi I'm trying to create namespace in an rbd pool, but get operation not supported. This is on a 16.2.6 Cephadm installed on Ubuntu 20.04.3. The pool is erasure encoded and the commands I run was the following. cephadm shell ceph osd pool create rbd 32 32 erasure ec42-jerasure-blaum_roth-hdd

[ceph-users] Re: empty lines in radosgw-admin bucket radoslist (octopus 15.2.16)

After removing some orphan objects (4million) I pulled the radoslist again and got the exact same files with the empty line between them. Can filenames contain a newline / cr character so the radosgw-admin tool just makes a new line in the output? Am Mi., 9. März 2022 um 17:50 Uhr schrieb Boris B