[ceph-users] Debian 12 support

Hi to all ceph community. I have a question about Debian 12 support for ceph 17. I didn’t find repo for that release at https://download.ceph.com/, does it supports and how to install ceph Quincy on Debian 12 ? Thank you to those who will answer. ___ c

[ceph-users] Re: Moving devices to a different device class?

Hi, well I will first check crush rules if device class is not defined there. If it is, then you have to create new crush rule and set it to the affected pools. dp On 10/26/23 23:36, Matt Larson wrote: Thanks Janne, It is good to know that moving the devices over to a new class is a safe

[ceph-users] ceph fs (meta) data inconsistent

Dear fellow cephers, today we observed a somewhat worrisome inconsistency on our ceph fs. A file created on one host showed up as 0 length on all other hosts: [user1@host1 h2lib]$ ls -lh total 37M -rw-rw 1 user1 user1 12K Nov 1 11:59 dll_wrapper.py [user2@host2 h2lib]# ls -l total 34 -rw-

[ceph-users] Ceph Leadership Team Meeting: 2023-11-1 Minutes

quincy 17.2.7: released! * major 'dashboard v3' changes causing issues? https://github.com/ceph/ceph/pull/54250 did not merge for 17.2.7 * planning a retrospective to discuss what kind of changes should go in minor releases when members of the dashboard team are present reef 18.2.1: * most PRs alr

[ceph-users] Re: ceph fs (meta) data inconsistent

We have seen issues like this a few times and they have all been kernel client bugs with CephFS’ internal “capability” file locking protocol. I’m not aware of any extant bugs like this in our code base, but kernel patches can take a long and winding path before they end up on deployed systems. Mos

[ceph-users] Re: "cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"

Hi, first of all, I'd still recommend to use the orchestrator to deploy OSDs. Building OSDs manually and then adopt them is redundant. Or do you have issues with the drivegroups? I don't have *the* solution but you could try to disable the mclock scheduler [1] which is the default since Qui

[ceph-users] Re: ceph orch problem

Hi, please provide more details about your cluster, especially the 'ceph -s' output. Is the cluster healthy? Apparently, other ceph commands work, but you could share the mgr logs anyway, maybe the hive mind finds something. ;-) Don't forget to mask sensitive data. Regards, Eugen Zitat v

[ceph-users] Re: Ceph OSD reported Slow operations

Hi, for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is

[ceph-users] Setting S3 bucket policies with multi-tenants

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any

[ceph-users] Re: Setting S3 bucket policies with multi-tenants

To update my own question, it would seem that Principle should be defined like this: - "Principal": {"AWS": ["arn:aws:iam::Tenant1:user/readwrite"]} And resource should: "Resource": [ "arn:aws:s3:::backups"] Is it worth having the docs updates - https://docs.ceph.com/en/quincy/radosgw/bu

[ceph-users] Re: ceph orch problem

Actually the cluster is in an error state due to (I think) these problems. ceph -s cluster: id: lksdjf health: HEALTH_ERR 18 failed cephadm daemon(s) 2 filesystems are degraded 1 filesystem has a failed mds daemon 1 filesystem is offlin

[ceph-users] Nautilus: Decommission an OSD Node

Hello., I've recently made the decision to gradually decommission my Nautilus cluster and migrate the hardware to a new Pacific or Quincy cluster. By gradually, I mean that as I expand the new cluster I will move (copy/erase) content from the old cluster to the new, making room to decommission mor

[ceph-users] negative list operation causing degradation in performance

In Ceph version 17.2.5, a notable performance drop is observed when executing list operations on non-existent objects, especially for large buckets undergoing high concurrency with frequent updates. For the context, initiating concurrent PUT/DELETE/LIST operations on 4 buckets, each containing

[ceph-users] Re: "cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"

> first of all, I'd still recommend to use the orchestrator to deploy OSDs. > Building > OSDs manually and then adopt them is redundant. Or do you have issues with > the drivegroups? I am having to do it this way because I couldn't find any doco on how to specify a separate DB/WAL device when

[ceph-users] Re: Ceph OSD reported Slow operations

Hi Eugen Please find the details below root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr:

[ceph-users] Re: Ceph OSD reported Slow operations

I'm afraid you're simply hitting the I/O limits of your disks. /Z On Thu, 2 Nov 2023 at 03:40, V A Prabha wrote: > Hi Eugen > Please find the details below > > > root@meghdootctr1:/var/log/ceph# ceph -s > cluster: > id: c59da971-57d1-43bd-b2b7-865d392412a5 > health: HEALTH_WARN > nodeep-scrub

[ceph-users] upgrade 17.2.6 to 17.2.7 , any issues?

Hello! I want to do upgrade from 17.2.6  to 17.2.7 , all 3 servers run alma 9. Could you share your experience? Any issues? Thank you! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 17.2.7 quincy dashboard issues

> > We have 4 ceph clusters going into the same prometheus instance. > Just curious, In the prometheus, if you want to see the details for a single cluster, how's it done through query? For reference, these are the queries that we are currently using now. USEDCAPACITY = 'ceph_cluster_total_used

[ceph-users] Re: Ceph OSD reported Slow operations

Can you please elaborate your identifications and the statement . On November 2, 2023 at 9:40 AM Zakhar Kirpichenko wrote: > I'm afraid you're simply hitting the I/O limits of your disks. > > /Z > > On Thu, 2 Nov 2023 at 03:40, V A Prabha < prab...@cdac.in > > wrot

[ceph-users] Re: Ceph OSD reported Slow operations

Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I assume size=3 replication so 1/3 of the total performance, and some 30%-ish OSD overhead. (36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very

[ceph-users] CephFS scrub causing MDS OOM-kill

Hi, I did setup CephFS forward scrub by executing cmd # ceph tell mds.cephfs:0 scrub start / recursive { "return_code": 0, "scrub_tag": "37a67f72-89a3-474e-8f8b-1e55cb979e14", "mode": "asynchronous" } But immediately after it started, memory usage on MDS that keeps rank 0 increased

[ceph-users] Re: Ceph OSD reported Slow operations

Thanks for your prompt reply .. But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server di