[ceph-users] Re: Move block.db to new ssd

Hi, For the record, we have identified the root cause of the overspilling issue. Previously, an ambiguity in the RocksDB resharding documentation led us to reshard our RocksDB databases using a lowercase 'o' instead of an uppercase 'O' in the command: $ cephadm shell --name osd.OSD_ID --fsid $

[ceph-users] Re: Sometimes PGs inconsistent (although there is no load on them)

This sounds like you have widespread inconsistencies that are surfaced by scrubs, not caused by them. Frequent causes: * Using a RAID HBA with bugs (all of them, in my experience), with broken preserved cache replay, with forced writeback cache without BBU, etc. * Power was out longer than BB

[ceph-users] Re: Ceph with 3 nodes and hybrid storage policy: how to configure OSDs with different HDD and SSD sizes

> > Hi, > > I want to setup a 3-node Ceph cluster with fault domain configured to "host". > > Each node should be equipped with: > > 6x SAS3 HDD 12TB > 1x SAS3 SSD 7TB (should be extended to 2x7 later) Is this existing hardware you’re stuck with? If not, don’t waste your money with SAS. S

[ceph-users] Re: Experience with 100G Ceph in Proxmox

Ah, my fault. On my phone the first search hit implied that this SKU was the S4520. > On Mar 11, 2025, at 10:17 AM, Alexander Schreiber > wrote: > > On Tue, Mar 11, 2025 at 08:50:55AM -0400, Anthony D'Atri wrote: >> Agree that the net is likely not your problem, though you should use iftop >

[ceph-users] Re: Ceph with 3 nodes and hybrid storage policy: how to configure OSDs with different HDD and SSD sizes

Hi Daniel, the first thing to mention is, while min_size 2/size 3 is good, having only three nodes leaves the cluster without any options to recover in case of a node failure. So it 's recommended to use at least four nodes. You have to be aware that the hybrid rule only gives you performan

[ceph-users] Re: Ceph cluster unable to read/write data properly and cannot recover normally.

We adjusted the pg_num and pgp_num of the .rgw.root storage pool from 8 to 128, and the cluster has been in data synchronization and balancing after the adjustment, and after about 1 day, the cluster became like this. ___ ceph-users mailing list -- ceph

[ceph-users] Re: Sometimes PGs inconsistent (although there is no load on them)

Hi Marianne, Am 3/11/25 um 11:24 schrieb Marianne Spiller: But nearly every day, one or two PGs get damaged. Never on the same OSDs. And there is no traffic on the storage as the virtualization hosts are not running. I see no further reason in the logs: everything is fine, scrub starts and l

[ceph-users] Sometimes PGs inconsistent (although there is no load on them)

Dear list, I'm currently maintaining several Ceph (prod) installations. One of them consists of 3 MON hosts and 6 OSD hosts hosting 40 OSDs in total. And there are 5 separate Proxmox-Hosts - they only host the VMs and use the storage provided by Ceph, but they are not part of Ceph. The worst c

[ceph-users] Re: Sometimes PGs inconsistent (although there is no load on them)

Hi,I suspect a hw issue. Please check the networks.Regards --martinAm 11.03.2025 11:24 schrieb Marianne Spiller :Dear list, I'm currently maintaining several Ceph (prod) installations. One of them consists of 3 MON hosts and 6 OSD hosts hosting 40 OSDs in total. And there are 5 separate Proxmox-

[ceph-users] Re: Move block.db to new ssd

Hi Robert, Thanks for pointing that out. This issue stems from version differences between our cluster environments: Our current Pacific cluster (migrating to Reef next week) uses the default configuration: bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P'

[ceph-users] Re: Experience with 100G Ceph in Proxmox

Hi Giovanna, El 11/3/25 a las 11:55, Giovanna Ratini escribió: We are running Ceph in Proxmox with a 10G network. Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive i

[ceph-users] Experience with 100G Ceph in Proxmox

Hello everyone, We are running Ceph in Proxmox with a 10G network. Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link

[ceph-users] Created no osd(s) on host, already created?

Hello list, We have a Ceph cluster with two management nodes and six data nodes. Each data node has 28 HDD disks. One disk recently failed in one of the nodes, corresponding to osd.2. To replace the disk, we took the osd.2 out, stopped it, and after a few days removed it, basically:   ceph

[ceph-users] Ceph with 3 nodes and hybrid storage policy: how to configure OSDs with different HDD and SSD sizes

Hi, I want to setup a 3-node Ceph cluster with fault domain configured to "host". Each node should be equipped with: 6x SAS3 HDD 12TB 1x SAS3 SSD 7TB (should be extended to 2x7 later) The ceph configuration should be size=3, min_size=2. All nodes are connected with 2x10Gbit (LACP). I want

[ceph-users] Re: RGW multisite metadata sync issue

Hi Vahideh, I’m facing the same issue. Could you please share more details on your solution? What do you mean by "changing the master zone" ? Thanks! Mahnoosh On Sat, 25 Jan 2025, 05:48 Vahideh Alinouri, wrote: > The metadata sync issue has been resolved by changing the master zone and > re-ru

[ceph-users] Re: Experience with 100G Ceph in Proxmox

On Tue, Mar 11, 2025 at 08:50:55AM -0400, Anthony D'Atri wrote: > Agree that the net is likely not your problem, though you should use iftop et > al to look for saturation. > > Check that you have proper smith hash policy, otherwise you may not be using > both bond links. > > The linked thr

[ceph-users] Re: Experience with 100G Ceph in Proxmox

Agree that the net is likely not your problem, though you should use iftop et al to look for saturation. Check that you have proper smith hash policy, otherwise you may not be using both bond links. The linked thread mentions SSDPE2KE032T8 which is SATA. The replication network is always

[ceph-users] Re: Move block.db to new ssd

Hi, Am 3/11/25 um 11:33 schrieb Frédéric Nass: $ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard As a consequence, the 'Object' key/value pairs were st

[ceph-users] Re: Ceph cluster unable to read/write data properly and cannot recover normally.

Can you show the output of 'ceph osd df'? Zitat von 13071007...@163.com: After the adjustment, the RGW service cannot be started, and the cluster and data cannot be restored to the normal state. The rgw service startup prompts an initialization timeout __

[ceph-users] Re: Module 'devicehealth' has failed

Hm, is it really necessary to configure all the device paths manually? I'd recommend to use rotational flags to distinguish between OSD and DB/WAL devices. Can you give it a try with a simpler spec file? Something like: service_type: osd service_id: node1.ec.all_disks service_name: osd.node