[ceph-users] Re: immutable bit

Unfortunately I think this ticket got put on the backburner then forgotten. I've asked the team if anyone wants to work on it. On Fri, Jul 7, 2023 at 6:38 PM Angelo Höngens wrote: > > Hey guys and girls, > > I noticed CephFS on my kinda default 17.2.6 CephFS volume, it does not > support setting

[ceph-users] immutable bit

Hey guys and girls, I noticed CephFS on my kinda default 17.2.6 CephFS volume, it does not support setting the immutable bit. (Want to start using it with the Veeam hardened repo that uses the immutable bit). I do see a lot of very, very old posts with technical details on how to implement it, bu

[ceph-users] Are replicas 4 or 6 safe during network partition? Will there be split-brain?

Hi, I'm evaluating if an even number of replicas is safe. Are 4 and 6 replicas still safe compared to replicas 5 and 7? With replicas 4 the min_size is 2. Does this mean in a split-brain situation there's a possibility that OSDs on both sides of the split could accept writes? Here's the scenari

[ceph-users] Re: Cannot get backfill speed up

Hello Paul, On Fri, Jul 7, 2023 at 5:13 PM Paul Mezzanini wrote: > I recently got mclock going literally an order of magnitude faster. I > would love to claim I found all the options myself but I collected the > knowledge of what knobs I needed to turn from here. > Significant usability and de

[ceph-users] Re: RGW dynamic resharding blocks write ops

We had a quite small window yesterday to debug, I found the error messages but we didn't collect the logs yet, I will ask them to do that on Monday. I *think* the error was something like this: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is s

[ceph-users] Re: RGW dynamic resharding blocks write ops

while a bucket is resharding, rgw will retry several times internally to apply the write before returning an error to the client. while most buckets can be resharded within seconds, very large buckets may hit these timeouts. any other cause of slow osd ops could also have that effect. it can be hel

[ceph-users] Re: MDSs report slow metadata IOs

https://docs.ceph.com/en/quincy/cephfs/createfs/#creating-pools As an additional note: it's recommended to put the metadata pool on a dedicated set of SSDs to prevent client load from disrupting MDS performance. On Fri, Jul 7, 2023 at 4:56 AM Ben wrote: > > Hi, > > see many of this in cluster lo

[ceph-users] Re: RGW dynamic resharding blocks write ops

That sounds reasonable, I will keep that in mind and discuss it with them, maybe they can establish something similar with their clients. Thanks! Zitat von "Szabo, Istvan (Agoda)" : I do manual reshard if needed but try to do pre-shard in advance. I try to deal with the user and ask them b

[ceph-users] Re: Cannot get backfill speed up

I recently got mclock going literally an order of magnitude faster. I would love to claim I found all the options myself but I collected the knowledge of what knobs I needed to turn from here. Steps I took: - Cleared all osd specific osd_mclock_max_capacity_iops settings. The auto created one

[ceph-users] Re: RGW dynamic resharding blocks write ops

I do manual reshard if needed but try to do pre-shard in advance. I try to deal with the user and ask them before onboard them, do they need bucket with more than a million objects (default 11 shard) or it’s enough. If they need I preshard (to a prime numbered shard number), if not then stay wit

[ceph-users] Re: RGW dynamic resharding blocks write ops

Okay, thanks for the comment. But does that mean that you never reshard or do you manually reshard? Do you experience performance degradation? Maybe I should also add that they have their index pool on HDDs (with rocksdb on SSD), not sure how big the impact is during resharding though. Zi

[ceph-users] Re: RGW dynamic resharding blocks write ops

I turned off :) Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- On 2023. Jul 7., at 17:35, Eugen Block wrote: Ema

[ceph-users] RGW dynamic resharding blocks write ops

Hi *, last week I successfully upgraded a customer cluster from Nautilus to Pacific, no real issues, their main use is RGW. A couple of hours after most of the OSDs were upgraded (the RGWs were not yet) their application software reported an error, it couldn't write to a bucket. This error

[ceph-users] Re: MON sync time depends on outage duration

We did look at the iostats of the disk, it was not saturated, but I don't have any specific numbers right now as I don't have direct access. But I'm open for more theories why waiting for 5 minutes lets the MON sync immediately but waiting for more takes so much more time. If necessary we'l

[ceph-users] MDSs report slow metadata IOs

Hi, see many of this in cluster log channel. many are blocked with long period of seconds. It should hurt client access performance. Any ideas to get rid of them? Thanks, Ben - 7/7/23 4:48:50 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_M

[ceph-users] Re: MON sync time depends on outage duration

This is a guess, the databases is like to swell. Especially the Level DB's, can grow x2 and reduce tens of percent of total size. This may be just another SST file creation, 1GB by default, Ii I remember it right Do you was looks to Grafana, about this HDD's utilization, IOPS? k Sent from my iP

[ceph-users] Re: MON sync time depends on outage duration

I forgot to add one question. @Konstantin, you wrote: I think that from the 3rd time the database just goes into compaction maintenance Can you share some more details what exactly you mean? Do you mean that if I restart a MON three times it goes into compaction maintenance and that it's

[ceph-users] Re: MON sync time depends on outage duration

Thanks, Dan! Yes that sounds familiar from the luminous and mimic days. The workaround for zillions of snapshot keys at that time was to use: ceph config set mon mon_sync_max_payload_size 4096 I actually did search for mon_sync_max_payload_keys, not bytes so I missed your thread, it seems