[ceph-users] Re: MDS crash

2024-04-22 Thread Alexey GERASIMOV
Ok, we will create the ticket. Eugen Block - ceph tell command needs to communicate with the MDS daemon running, but it is crashed. So, I just have the information about the impossibility to receive the information from daemon: ceph tell mds.0 damage ls Error ENOENT: problem getting command de

[ceph-users] Re: MDS crash

2024-04-22 Thread Eugen Block
Right, I just figured from the health output you would have a couple of seconds or so to query the daemon: mds: 1/1 daemons up Zitat von Alexey GERASIMOV : Ok, we will create the ticket. Eugen Block - ceph tell command needs to communicate with the MDS daemon running, but it is cr

[ceph-users] Re: RGWs stop processing requests after upgrading to Reef

2024-04-22 Thread Eugen Block
Hi, I don't see a reason why Quincy rgw daemons shouldn't work with a Reef cluster. It would basically mean that you have a staggered upgrade [1] running and didn't upgrade RGWs yet. It should also work to just downgrade them, either by providing a different default image, then redeploy r

[ceph-users] Re: RGWs stop processing requests after upgrading to Reef

2024-04-22 Thread Eugen Block
Sorry, I missed the "client" entity: host01:~ # ceph config set client container_image my-registry:ceph/ceph:v16.2.13.66 host01:~ # ceph orch redeploy my-rgw Now I have mix versions: host01:~ # ceph versions -f json | jq '.rgw' { "ceph version 16.2.13-66-g54799ee0666 (54799ee06669271880e

[ceph-users] rbd-mirror failed to query services: (13) Permission denied

2024-04-22 Thread Stefan Kooman
Hi, We are testing rbd-mirroring. There seems to be a permission error with the rbd-mirror user. Using this user to query the mirror pool status gives: failed to query services: (13) Permission denied And results in the following output: health: UNKNOWN daemon health: UNKNOWN image health: O

[ceph-users] Stuck in replay?

2024-04-22 Thread Erich Weiler
Hi All, We have a somewhat serious situation where we have a cephfs filesystem (18.2.1), and 2 active MDSs (one standby). ThI tried to restart one of the active daemons to unstick a bunch of blocked requests, and the standby went into 'replay' for a very long time, then RAM on that MDS serve

[ceph-users] Re: Stuck in replay?

2024-04-22 Thread Erich Weiler
I also see this from 'ceph health detail': # ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 MDSs report oversized cache; 1 MDSs behind on trimming [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.s

[ceph-users] Re: Stuck in replay?

2024-04-22 Thread Sake Ceph
Just a question: is it possible to block or disable all clients? Just to prevent load on the system. Kind regards, Sake > Op 22-04-2024 20:33 CEST schreef Erich Weiler : > > > I also see this from 'ceph health detail': > > # ceph health detail > HEALTH_WARN 1 filesystem is degraded; 1 MDSs

[ceph-users] Re: Stuck in replay?

2024-04-22 Thread Erich Weiler
possibly but it would be pretty time consuming and difficult... Is it maybe a RAM issue since my MDS RAM is filling up? Should maybe I bring up another MDS on another server with huge amount of RAM and move the MDS there in hopes it will have enough RAM to complete the replay? On 4/22/24 11:

[ceph-users] Re: Stuck in replay?

2024-04-22 Thread Eugen Block
IIRC, you have 8 GB configured for the mds cache memory limit, and it doesn’t seem to be enough. Does the host run into oom killer as well? But it’s definitely a good approach to increase the cache limit (try 24 GB if possible since it’s trying to use at least 19 GB) on a host with enough R

[ceph-users] Re: Stuck in replay?

2024-04-22 Thread Erich Weiler
I was able to start another MDS daemon on another node that had 512GB RAM, and then the active MDS eventually migrated there, and went through the replay (which consumed about 100 GB of RAM), and then things recovered. Phew. I guess I need significantly more RAM in my MDS servers... I had no

[ceph-users] Re: Stuck in replay?

2024-04-22 Thread Sake Ceph
100 GB of Ram! Damn that's a lot for a filesystem in my opinion, or am I wrong? Kind regards, Sake > Op 22-04-2024 21:50 CEST schreef Erich Weiler : > > > I was able to start another MDS daemon on another node that had 512GB > RAM, and then the active MDS eventually migrated there, and wen

[ceph-users] Re: Question about PR merge

2024-04-22 Thread Xiubo Li
Hi Nigel, For your issue I created a dedicated tracker, please see https://tracker.ceph.com/issues/65630. I have found the root cause and I am still trying to find the proper way to fix it. Please watch the tracker. Thanks - Xiubo On 4/18/24 14:22, Nigel Williams wrote: Hi Xiubo, Is the

[ceph-users] ceph api rgw/role

2024-04-22 Thread farhad kh
hi , i used ceph api for create rgw/role but ther is not api for delete or edit rgw/role . how can i delete them or edit ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph api rgw/role

2024-04-22 Thread Pritha Srivastava
There are REST APIs 1. UpdateRole to modify a role 2. UpdateAssumeRolePolicy to modify the trust policy of a role. What exactly are you trying to modify for a role? Thanks, Pritha On Tue, Apr 23, 2024 at 11:00 AM farhad kh wrote: > hi , i used ceph api for create rgw/role but ther is not api

[ceph-users] Re: Status of IPv4 / IPv6 dual stack?

2024-04-22 Thread Zac Dover
I have removed dual-stack-mode-related information from the documentation on the assumption that dual-stack mode was planned but never fully implemented. See https://tracker.ceph.com/issues/65631. See https://github.com/ceph/ceph/pull/57051. Hat-tip to Dan van der Ster, who bumped this thread