[ceph-users] Accessing OSD objects

2023-02-24 Thread Geoffrey Rhodes
Hi Anthony, thanks for reaching out. Erasure data pool (K=4, M=2) but I had more than two disk failures around the same time and the data had not fully replicated / restored elsewhere in the cluster. They are big 12TB Exos so it usually takes a few weeks to backfill / recover plus I had snaptrimm

[ceph-users] Large STDDEV in pg per osd

2023-02-24 Thread Joe Ryner
I have been digging for a while on how to minimize STDDEV of the distribution of data on my OSDs and I can't seem to get it to get below 12. I have other clusters that have a STDDEV of 1 which is my goal. But this cluster is really giving me fits. This cluster started off on the Emperor. Might

[ceph-users] OpenSSL in librados

2023-02-24 Thread Patrick Schlangen
Hi, please forgive me if this has been asked before - I could not find any information on this topic. I am using ceph with librados via the phprados extension. Since upgrading to the current ceph versions where OpenSSL is used in in librados, I observe that using PHP's libcurl integration and

[ceph-users] Re: mons excessive writes to local disk and SSD wearout

2023-02-24 Thread Dan van der Ster
Hi Andrej, That doesn't sound right -- I checked a couple of our clusters just now and the mon filesystem is writing at just a few 100kBps. debug_mon = 10 should clarify the root cause. Perhaps it's logm from some persistent slow ops? Cheers, Dan On Fri, Feb 24, 2023 at 7:36 AM Andrej Filipci

[ceph-users] Accessing OSD objects

2023-02-24 Thread Geoffrey Rhodes
Hello all, I'd really appreciate some input from the more knowledgeable here. Is there a way I can access OSD objects if I have a BlueFS replay error? This error prevents me starting the OSD and also throws an error if I try using the bluestore or objectstore tools. - I can however run a ceph-blu

[ceph-users] Re: rbd map error: couldn't connect to the cluster!

2023-02-24 Thread Thomas Schneider
Please check the output here: # rbd info hdb_backup/VCT rbd image 'VCT':     size 800 GiB in 204800 objects     order 22 (4 MiB objects)     snapshot_count: 0     id: b768d4baac048b     block_name_prefix: rbd_data.b768d4baac048b     format: 2     features: layering

[ceph-users] mons excessive writes to local disk and SSD wearout

2023-02-24 Thread Andrej Filipcic
Hi, on our large ceph cluster with 60 servers, 1600 OSDs, we have observed that small system nvmes are wearing out rapidly. Our monitoring shows mon writes on average about 10MB/s to store.db. For small system nvmes of 250GB and DWPD of ~1, this turns out to be too much, 0.8TB/day or 1.5PB i

[ceph-users] Re: rbd map error: couldn't connect to the cluster!

2023-02-24 Thread Eugen Block
Just one addition from my test, I believe I misinterpreted my results because my test image was named "test" and the client "TEST", so the rbd_id. is indeed upper case for an image that has an upper-case name. So forget my comment about that. Another question though: does the image you're try

[ceph-users] Re: rbd map error: couldn't connect to the cluster!

2023-02-24 Thread Thomas Schneider
Actually I didn't try other caps. The setup of RBD images and authorizations is automised with a bash script that worked in the past w/o issues. I need to understand the root cause in order to adapt the script accordingly. Am 23.02.2023 um 17:55 schrieb Eugen Block: And did you already try