[ceph-users] Ceph Logging Configuration and "Large omap objects found"

Hi, I have a bunch of long-standing struggles with the way Ceph handles logging and I cannot figured out how to solve them. These issues are basically the following: - The log config options are utterly confusing and very badly documented - Mon file logs are spammed with DBG-level cluster log

[ceph-users] Re: Ceph Logging Configuration and "Large omap objects found"

Thanks all. ceph log last 10 warn cluster That outputs nothing for me. Any docs about this? I don't have much to comment about logging, I feel you though. I just wanted to point out that the details about the large omap object should be in the (primary) OSD log, not in the MON log: The me

[ceph-users] Re: Ceph Logging Configuration and "Large omap objects found"

That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something? BTW, which ceph version are we talking about here? reef. I tried ceph log last 100 debug cluster and that gives me the usual DBG

[ceph-users] Re: Ceph Logging Configuration and "Large omap objects found"

take a closer look if and how that can be handled properly. Zitat von Janek Bevendorff : That's where the 'ceph log last' commands should help you out, but I don't know why you don't see it, maybe increase the number of lines to display or something? BTW, which

[ceph-users] Re: Ceph Logging Configuration and "Large omap objects found"

sts with a command? Zitat von Janek Bevendorff : Thanks. I increased the number even further and got a (literal) handful of non-debug messages. Unfortunately, none were relevant for the problem I'm trying to debug. On 13/08/2024 14:03, Eugen Block wrote: Interesting, apparently the number

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

We are having the exact same problem (also Octopus). The object is listed by s3cmd, but trying to download it results in a 404 error. radosgw-admin object stat shows that the object still exists. Any further ideas how I can restore access to this object? (Sorry if this is a duplicate, but it s

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

netflix_mp4' {tmpfile} The first grep is actually the S3 multipart ID string added to the prefix by rgw. Rafael On Tue, 10 Nov 2020 at 01:04, Janek Bevendorff <mailto:janek.bevendo...@uni-weimar.de>> wrote: We are having the exact same problem (also Octopus). The object is

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

me.19: (2) No such file or directory On 10/11/2020 10:14, Janek Bevendorff wrote: Thanks for the reply. This issue seems to be VERY serious. New objects are disappearing every day. This is a silent, creeping data loss. I couldn't find the object with rados stat, but I am now listin

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

Here's something else I noticed: when I stat objects that work via radosgw-admin, the stat info contains a "begin_iter" JSON object with RADOS key info like this "key": { "name": "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.g

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

We are having the exact same problem (also Octopus). The object is listed by s3cmd, but trying to download it results in a 404 error. radosgw-admin object stat shows that the object still exists. Any further ideas how I can restore access to this object? _

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

s (eg. renaming/deleting). I had thought perhaps it was a bug with the rgw garbage collector..but that is pure speculation. Once you can articulate the problem, I'd recommend logging a bug tracker upstream. On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff <mailto:janek.bevendo...@uni-

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

Here is a bug report concerning (probably) this exact issue: https://tracker.ceph.com/issues/47866 I left a comment describing the situation and my (limited) experiences with it. On 11/11/2020 10:04, Janek Bevendorff wrote: Yeah, that seems to be it. There are 239 objects prefixed

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

huxia...@horebdata.cn *From:* EDH - Manuel Rios <mailto:mrios...@easydatahost.com> *Date:* 2020-11-12 14:27 *To:* Janek Bevendorff <mailto:janek.bevendo...@uni-weimar.de>; Rafael Lopez <m

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

ery dangerous bug for data safety. Hope the bug would be quickly identified and fixed. best regards, Samuel huxia...@horebdata.cn <mailto:huxia...@horebdata.cn> From: Janek Bevendorff Date: 2020-11-12 18:17 To:huxia...@horebdata.cn <mailto:huxia...@horebdata.cn>; EDH - Manuel Ri

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

comments there, so everything’s together. I do want to follow up on your answer to Question 4, Janek: On Nov 13, 2020, at 12:22 PM, Janek Bevendorff <mailto:janek.bevendo...@uni-weimar.de>> wrote: 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

om/issues/47866 We may want to move most of the conversation to the comments there, so everything’s together. I do want to follow up on your answer to Question 4, Janek: On Nov 13, 2020, at 12:22 PM, Janek Bevendorff <mailto:janek.bevendo...@uni-weimar.de>> wrote: 4. Is anyone exper

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

FYI: I have radosgw-admin gc list --include-all running every three minutes for a day, but the list has stayed empty. Though I haven't seen any further data loss, either. I will keep it running until the next time I seen an object vanish. On 17/11/2020 09:22, Janek Bevendorff wrote: I

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

The object, a Docker layer, that went missing has not been touched in 2 months. It worked for a while, but then suddenly went missing. Was the object a multipart object? You can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns": "multipar

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the "object" there) On 18/11/2020 09:58, Janek Bevendorff wrote: The object, a Docker layer, that went missing has not been touched in 2 months. It worked for a while, but then suddenly we

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

about Rados, so I’m not sure this is helpful. On 18 Nov 2020, at 10:01, Janek Bevendorff wrote: Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the "object" there) On 18/11/2020 09:58, Janek Bevendorff wrote: The object, a Docker layer, th

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

m not knowledgable about Rados, so I’m not sure this is helpful. On 18 Nov 2020, at 10:01, Janek Bevendorff wrote: Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the "object" there) On 18/11/2020 09:58, Janek Bevendorff wrote: The object,

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

try:     s3.Object(bucket, obj_name).get()['Body'].read(1)     return False     except ClientError:     return True With 600 executors on 130 hosts, it takes about 30 seconds for a 300k object bucket. On 19/11/2020 09:21, Janek Bevendorff wrote: I would recommend you get

[ceph-users] Re: MDS lost, Filesystem degraded and wont mount

This is very common issue. Deleting mdsX_openfiles.Y has become part of my standard maintenance repertoire. As soon as you have a few more clients and one of them starts opening and closing files in rapid succession (or does other metadata-heavy things), it becomes very likely that the MDS cras

[ceph-users] Re: MDS lost, Filesystem degraded and wont mount

laced by a standby before it's finished). I guess we're pretty lucky with our CephFS's because we have more than 1k clients and it is pretty solid (though the last upgrade had a hiccup decreasing down to single active MDS). -- Dan On Fri, Dec 4, 2020 at 8:20 PM Janek Bevendorff

[ceph-users] Re: MDS lost, Filesystem degraded and wont mount

few hungry GPUs. -- Dan I guess we're pretty lucky with our CephFS's because we have more than 1k clients and it is pretty solid (though the last upgrade had a hiccup decreasing down to single active MDS). -- Dan On Fri, Dec 4, 2020 at 8:20 PM Janek Bevendorff wrote: This

[ceph-users] Re: Provide more documentation for MDS performance tuning on large file systems

Thanks, Dan! I have played with many thresholds, including the decay rates. It is indeed very difficult to assess their effects, since our workloads differ widely depending on what people are working on at the moment. I would need to develop a proper benchmarking suite to simulate the differe

[ceph-users] Re: Provide more documentation for MDS performance tuning on large file systems

What exactly do you set to 64k? We used to set mds_max_caps_per_client to 5, but once we started using the tuned caps recall config, we reverted that back to the default 1M without issue. mds_max_caps_per_client. As I mentioned, some clients hit this limit regularly and they aren't entir

[ceph-users] Re: Provide more documentation for MDS performance tuning on large file systems

for a few sub trees. Despite mds_export_ephemeral_distributed being set to true, all work is done by mds.0 now and I also don't see any additional pins in ceph tell mds.\* get subtrees. Any ideas why that might be? On 07/12/2020 10:49, Dan van der Ster wrote: On Mon, Dec 7, 2020 at 10:3

[ceph-users] Re: Provide more documentation for MDS performance tuning on large file systems

Never mind, when I enable it on a more busy directory, I do see new ephemeral pins popping up. Just not on the directories I set it on originally. Let's see how that holds up. On 07/12/2020 13:04, Janek Bevendorff wrote: Thanks. I tried playing around a bit

[ceph-users] Re: MDS lost, Filesystem degraded and wont mount

This sounds like there is one or a few clients acquiring too many caps. Have you checked this? Are there any messages about the OOM killer? What config changes for the MDS have you made? Yes, it's individual clients acquiring too my caps. I first ran the adjusted recall settings you suggeste

[ceph-users] Re: Provide more documentation for MDS performance tuning on large file systems

Hi Patrick, I haven't gone through this thread yet but I want to note for those reading that we do now have documentation (thanks for the frequent pokes Janek!) for the recall configurations: https://docs.ceph.com/en/latest/cephfs/cache-configuration/#mds-recall Please let us know if it's missi

[ceph-users] Re: MDS lost, Filesystem degraded and wont mount

Wow! Distributed epins :) Thanks for trying it. How many sub-directories under the distributed epin'd directory? (There's a lot of stability problems that are to be fixed in Pacific associated with lots of subtrees so if you have too large of a directory, things could get ugly!) Yay, beta tes

[ceph-users] Re: mgr's stop responding, dropping out of cluster with _check_auth_rotating

Do you have the prometheus module enabled? Turn that off, it's causing issues. I replaced it with another ceph exporter from Github and almost forgot about it. Here's the relevant issue report: https://tracker.ceph.com/issues/39264#change-179946 On 10/12/2020 16:43, Welby McRoberts wrote: H

[ceph-users] Re: mgr's stop responding, dropping out of cluster with _check_auth_rotating

FYI, this is the ceph-exporter we're using at the moment: https://github.com/digitalocean/ceph_exporter It's not as good, but it does the job mostly. Some more specific metrics are missing, but the majority is there. On 10/12/2020 19:01, Janek Bevendorff wrote: Do you have the

[ceph-users] Re: Provide more documentation for MDS performance tuning on large file systems

My current settings are: mds   advanced  mds_beacon_grace 15.00 mds   basic mds_cache_memory_limit 4294967296 mds   advanced  mds_cache_trim_threshold 393216 global    advanced  mds_export_ephemeral_distributed true mds   advanced  mds_recall_global_max

[ceph-users] Re: Provide more documentation for MDS performance tuning on large file systems

My current settings are: mds advanced mds_beacon_grace 15.00 True. I might as well remove it completely, it's an artefact of earlier experiments. This should be a global setting. It is used by the mons and mdss. mds basic mds_cache_memory_limit 4294967296

[ceph-users] MON slow ops and growing MON store

Hi, All of a sudden, we are experiencing very concerning MON behaviour. We have five MONs and all of them have thousands up to tens of thousands of slow ops, the oldest one blocking basically indefinitely (at least the timer keeps creeping up). Additionally, the MON stores keep inflating heavil

[ceph-users] Re: MON slow ops and growing MON store

basic log_to_stderr false globalbasic log_to_syslog true globaladvanced mon_cluster_log_file_level error global

[ceph-users] Re: MON slow ops and growing MON store

or are they flapping with this load? > > .. dan > > > > On Thu, Feb 25, 2021, 8:58 PM Janek Bevendorff > mailto:janek.bevendo...@uni-weimar.de>> > wrote: > Thanks, Dan. > > On the first MON, the command doesn’t even retu

[ceph-users] Re: MON slow ops and growing MON store

ksDB stuff, but nothing that > > actually looks serious or even log-worthy. I noticed that before that > > despite logging being set to warning level, the cluster log keeps being > > written to the MON log. But it shouldn’t cause such massive stability > > issues, should it? The date

[ceph-users] Re: MON slow ops and growing MON store

Thanks, I’ll try that tomorrow. > On 25. Feb 2021, at 21:59, Dan van der Ster wrote: > > Maybe the debugging steps in that insights tracker can be helpful > anyway: https://tracker.ceph.com/issues/39955 > > -- dan > > On Thu, Feb 25, 2021 at 9:27 PM Janek Bevendorff

[ceph-users] Re: MON slow ops and growing MON store

> On 25. Feb 2021, at 22:17, Dan van der Ster wrote: > > Also did you solve your log spam issue here? > https://tracker.ceph.com/issues/49161 > Surely these things are related? No. But I noticed that DBG log spam only happens when log_to_syslog is enabled. systemd is smart enough to avoid fi

[ceph-users] Re: MON slow ops and growing MON store

fic reason for the incident yesterday in the logs besides a few more RocksDB status and compact messages than usual, but that's more symptomatic. On 26/02/2021 13:05, Mykola Golub wrote: On Thu, Feb 25, 2021 at 08:58:01PM +0100, Janek Bevendorff wrote: On the first MON, the command does

[ceph-users] PG inactive when host is down despite CRUSH failure domain being host

Hi, I am having a weird phenomenon, which I am having trouble to debug. We have 16 OSDs per host, so when I reboot one node, 16 OSDs will be missing for a short time. Since our minimum CRUSH failure domain is host, this should not cause any problems. Unfortunately, I always have handful (1-5)

[ceph-users] Re: PG inactive when host is down despite CRUSH failure domain being host

ize 2 which would explain what you're describing. Zitat von Janek Bevendorff : Hi, I am having a weird phenomenon, which I am having trouble to debug. We have 16 OSDs per host, so when I reboot one node, 16 OSDs will be missing for a short time. Since our minimum CRUSH failure domain is

[ceph-users] Re: MON slow ops and growing MON store

26/02/2021 15:24, Janek Bevendorff wrote: Since the full cluster restart and disabling logging to syslog, it's not a problem any more (for now). Unfortunately, just disabling clog_to_monitors didn't have the wanted effect when I tried it yesterday. But I also believe that it is somehow r

[ceph-users] Ceph orch OSD redeployment after boot on stateless RAM root

Hi, I recently moved from a manual Ceph deployment using Saltstack to a hybrid of Saltstack and cephadm / ceph orch. We are provisioning our Ceph hosts using a stateless PXE RAM root, so I definitely need Saltstack to bootstrap at least the Ceph APT repository and the MON/MGR deployment. Afte

[ceph-users] RadosGW load balancing with Kubernetes + ceph orch

Hey all, My Ceph cluster is managed mostly by cephadm / ceph orch to avoid circular dependencies between in our infrastructure deployment. Our RadosGW endpoints, however, are managed by Kubernetes, since it provides proper load balancing and service health checks. This leaves me in the unsat

[ceph-users] Redeploy ceph orch OSDs after reboot, but don't mark as 'unmanaged'

Hi, We have our cluster RAM-booted, so we start from a clean slate after every reboot. That means I need to redeploy all OSD daemons as well. At the moment, I run cephadm deploy via Salt on the rebooted node, which brings the deployed OSDs back up, but the problem with this is that the deploy

[ceph-users] Re: Redeploy ceph orch OSDs after reboot, but don't mark as 'unmanaged'

wouldn't be an option either, even if I were willing to deploy the admin key on the OSD hosts. On 07/11/2023 11:41, Janek Bevendorff wrote: Hi, We have our cluster RAM-booted, so we start from a clean slate after every reboot. That means I need to redeploy all OSD daemons as well.

[ceph-users] Re: Redeploy ceph orch OSDs after reboot, but don't mark as 'unmanaged'

host": ["XXX"], "target": ["mon-mgr", ""]} to mon-mgr Created no osd(s) on host XXX; already created? I suspect that it doesn't work for OSDs that are not explicitly marked as managed by ceph orch. But how do I do that? I also commented the tr

[ceph-users] Re: Redeploy ceph orch OSDs after reboot, but don't mark as 'unmanaged'

ame. I checked whether cephadm deploy perhaps has an undocumented flag for setting the service name, but couldn't find any. I could run deploy, change the service name and then restart the service, but that's quite ugly. Any better ideas? Janek Zitat von Janek Bevendorff : Hi Eugen

[ceph-users] MDS "newly corrupt dentry" after patch version upgrade

Hi, After a patch version upgrade from 16.2.10 to 16.2.12, our rank 0 MDS fails start start. After replaying the journal, it just crashes with [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) Immediately after the upgrade, I

[ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade

working on this issue. Cheers, Dan __ Clyso GmbH | https://www.clyso.com On Tue, May 2, 2023 at 7:32 AM Janek Bevendorff wrote: Hi, After a patch version upgrade from 16.2.10 to 16.2.12, our rank 0 MDS fails start start. After replaying the journal, it just crashes with [ERR] : MDS

[ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade

Hi Patrick, Please be careful resetting the journal. It was not necessary. You can try to recover the missing inode using cephfs-data-scan [2]. Yes. I did that very reluctantly after trying everything else as a last resort. But since it only gave me another error, I restored the previous sta

[ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade

Hi Patrick, I'll try that tomorrow and let you know, thanks! I was unable to reproduce the crash today. Even with mds_abort_on_newly_corrupt_dentry set to true, all MDS booted up correctly (though they took forever to rejoin with logs set to 20). To me it looks like the issue has resolved

[ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade

he output file is corrupt? Any way I can fix it? The output file has 14 million lines. We have about 24.5 million objects in the metadata pool. Janek On 03/05/2023 14:20, Patrick Donnelly wrote: On Wed, May 3, 2023 at 4:33 AM Janek Bevendorff wrote: Hi Patrick, I'll try that tomorrow an

[ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade

have some files with binary keys that cannot be decoded as utf-8. Unfortunately, the rados python library assumes that omap keys can be decoded this way. I have a ticket here: https://tracker.ceph.com/issues/59716 I hope to have a fix soon. On Thu, May 4, 2023 at 3:15 AM Janek Bevendorff

[ceph-users] CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of around 15TB which hosts our CephFS metadata pool. Usually, that's more than enough. The normal size of th

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Forgot to add: We are still on Nautilus (16.2.12). On 31/05/2023 09:53, Janek Bevendorff wrote: Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

618 flags hashpspool,nodelete stripe_width 0 expected_num_objects 300 recovery_op_priority 5 recovery_priority 2 application cephfs Janek On 31/05/2023 10:10, Janek Bevendorff wrote: Forgot to add: We are still on Nautilus (16.2.12). On 31/05/2023 09:53, Janek Bevendorff wrote: Hi, Per

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

r MDS, probably because it wasn't that extreme yet and I had reduced the maximum cache size. Still looks like a bug to me. On 31/05/2023 11:18, Janek Bevendorff wrote: Another thing I just noticed is that the auto-scaler is trying to scale the pool down to 128 PGs. That could also resul

[ceph-users] Re: MDS corrupt (also RADOS-level copy?)

Hi Jake, Very interesting. This sounds very much like what we have been experiencing the last two days. We also had a sudden fill-up of the metadata pool, which repeated last night. See my question here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7U27L27FHHPDYGA6VNNVWGLTXC

[ceph-users] Re: MDS corrupt (also RADOS-level copy?)

. None of these operations are safe, though. On 31/05/2023 16:41, Janek Bevendorff wrote: Hi Jake, Very interesting. This sounds very much like what we have been experiencing the last two days. We also had a sudden fill-up of the metadata pool, which repeated last night. See my question here

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

rs, Dan > > ______ > Clyso GmbH | https://www.clyso.com > > > On Wed, May 31, 2023 at 4:03 AM Janek Bevendorff > wrote: >> >> I checked our logs from yesterday, the PG scaling only started today, >> perhaps triggered by the snapshot tr

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

en at debug_mds = 20). Any idea what the reason could be? I checked whether they have enough RAM, which seems to be the case (unless they try to allocate tens of GB in one allocation). Janek On 31/05/2023 21:57, Janek Bevendorff wrote: Hi Dan, Sorry, I meant Pacific. The version number was co

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

nek [1] "Newly corrupt dentry" ML link: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/JNZ6V5WSYKQTNQPQPLWRBM2GEP2YSCRV/#PKQVZYWZCH7P76Q75D5WD5JEAVWOKJE3 [2] ceph-post-file ID: 7c039483-49fd-468c-ba40-fb10337aa7d6 On 05/06/2023 16:08, Janek Bevendorff wrote: I ju

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

That said, our MON store size has also been growing slowly from 900MB to 5.4GB. But we also have a few remapped PGs right now. Not sure if that would have an influence. On 05/06/2023 17:48, Janek Bevendorff wrote: Hi Patrick, hi Dan! I got the MDS back and I think the issue is connected to

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

store is full. On 05/06/2023 18:03, Janek Bevendorff wrote: That said, our MON store size has also been growing slowly from 900MB to 5.4GB. But we also have a few remapped PGs right now. Not sure if that would have an influence. On 05/06/2023 17:48, Janek Bevendorff wrote: Hi Patrick, hi

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

06/06/2023 09:16, Janek Bevendorff wrote: I checked our Prometheus logs and the number of log events of individual MONs are indeed randomly starting to increase dramatically all of a sudden. I attached a picture of the curves. The first incidence you see there was when our metadata store

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Hi Patrick, I'm afraid your ceph-post-file logs were lost to the nether. AFAICT, our ceph-post-file storage has been non-functional since the beginning of the lab outage last year. We're looking into it. I have it here still. Any other way I can send it to you? Extremely unlikely. Okay, ta

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

rt four days ago. I ran your ceph-gather.sh script a couple of times, but dumps only mds.0. Should I modify it to dump mds.3 instead so you can have a look? Janek On 10/06/2023 15:23, Patrick Donnelly wrote: On Fri, Jun 9, 2023 at 3:27 AM Janek Bevendorff wrote: Hi Patrick, I'm afraid y

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Hi Patrick, The event log size of 3/5 MDS is also very high, still. mds.1, mds.3, and mds.4 report between 4 and 5 million events, mds.0 around 1.4 million and mds.2 between 0 and 200,000. The numbers have been constant since my last MDS restart four days ago. I ran your ceph-gather.sh script a

[ceph-users] CephFS warning: clients laggy due to laggy OSDs

Hey all, Since the upgrade to Ceph 16.2.14, I keep seeing the following warning: 10 client(s) laggy due to laggy OSDs ceph health detail shows it as: [WRN] MDS_CLIENTS_LAGGY: 10 client(s) laggy due to laggy OSDs     mds.***(mds.3): Client *** is laggy; not evicted because some OSD(s) is/are l

[ceph-users] Re: CephFS warning: clients laggy due to laggy OSDs

able. On Mon, Sep 18, 2023 at 10:51 AM Janek Bevendorff wrote: Hey all, Since the upgrade to Ceph 16.2.14, I keep seeing the following warning: 10 client(s) laggy due to laggy OSDs ceph health detail shows it as: [WRN] MDS_CLIENTS_LAGGY: 10 client(s) laggy due to

[ceph-users] Re: CephFS warning: clients laggy due to laggy OSDs

that the affected hosts are usually the same, but I have absolutely no clue why. Janek On 19/09/2023 12:36, Venky Shankar wrote: Hi Janek, On Mon, Sep 18, 2023 at 9:52 PM Janek Bevendorff wrote: Thanks! However, I still don't really understand why I am seeing this. This is due to

[ceph-users] Re: CephFS warning: clients laggy due to laggy OSDs

ous email? On Wed, Sep 20, 2023 at 3:07 PM Venky Shankar wrote: > Hi Janek, > > On Tue, Sep 19, 2023 at 4:44 PM Janek Bevendorff < > janek.bevendo...@uni-weimar.de> wrote: > >> Hi Venky, >> >> As I said: There a

[ceph-users] Re: CephFS warning: clients laggy due to laggy OSDs

seems to do more harm than good. I can see why it's there, but the implementation appears to be rather buggy. I also set mds_session_blocklist_on_timeout to false, because I had the impression that clients where being blocklisted too quickly. On 21/09/2023 09:24, Janek Bevendorff wrote

[ceph-users] Re: pgs incossistent every day same osd

Yes. If you've seen this reoccur multiple times, you can expect it will only get worse with time. You should replace the disk soon. Very often these disks are also starting to slow down other operations in the cluster as the read times increase. On 26/09/2023 13:17, Jorge JP wrote: Hello, F

[ceph-users] Release Index and Docker Hub images outdated

Hi, The release index on docs.ceph.com is outdated by two Pacific patch releases: https://docs.ceph.com/en/quincy/releases/index.html The image tags on Docker Hub are even more outdated and stop at v16.2.5. quay.io seems to be up to date. Can these be updated please? Thanks Janek _

[ceph-users] Re: Release Index and Docker Hub images outdated

: https://docs.ceph.com/en/latest/install/containers/ > On 30. May 2022, at 14:47, Robert Sander wrote: > > Am 30.05.22 um 13:16 schrieb Janek Bevendorff: > >> The image tags on Docker Hub are even more outdated and stop at v16.2.5. >> quay.io seems to be up to date.

[ceph-users] Re: Release Index and Docker Hub images outdated

The quay.io/ceph/daemon:latest-pacific image is also stuck on 16.2.5. Only the quay.io/ceph/ceph:v16 image seems to be up to date, but I can't get it to start the daemons. On 30/05/2022 14:54, Janek Bevendorff wrote: Was this announced somewhere? Could this not wait till Pacific is EO

[ceph-users] Containerized radosgw crashes randomly at startup

Hi, This is an issue I've been having since at least Ceph 15.x and I haven't found a way around it yet. I have a bunch of radosgw nodes in a Kubernetes cluster (using the ceph/ceph-daemon Docker image) and once every few container restarts, the daemon decides to crash at startup for unknown r

[ceph-users] Re: Containerized radosgw crashes randomly at startup

ctory on all hosts and restarting the deployment, all pods came back up again. So I guess something was wrong with the keys stored on some of the host machines. Janek On 31/05/2022 11:08, Janek Bevendorff wrote: Hi, This is an issue I've been having since at least Ceph 15.x and I haven

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

ejoin, but it tells me that my settings do not fully solve the problem yet (unless perhaps I increase the trim threshold even further. On 06.08.19 19:52, Janek Bevendorff wrote: Your parallel rsync job is only getting 150 creates per second? What was the previous throughput? I am actually not

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

The fix has been merged into master and will be backported soon. Amazing, thanks! I've also done testing in a large cluster to confirm the issue you found. Using multiple processes to create files as fast as possible in a single client reliably reproduced the issue. The MDS cannot recall c

[ceph-users] Re: mgr daemons becoming unresponsive

These issues sound a bit like a bug I reported a few days ago: https://tracker.ceph.com/issues/39264 Related: https://tracker.ceph.com/issues/39264 On 02/11/2019 17:34, Oliver Freyermuth wr

[ceph-users] Re: mgr daemons becoming unresponsive

es seem to be triggered by the devicehealth module). Cheers, Oliver On 2019-11-02 18:28, Janek Bevendorff wrote: These issues sound a bit like a bug I reported a few days ago: https://tracker.ceph.com/issues/39264 <https://tracker.ceph.com/issues/39264#change-149689>

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

failing to rejoin). Or at least give us a setting for a clean MDS start similar to what deleting the mds0_openfiles objects does. Cheers Janek On 30/08/2019 09:27, Janek Bevendorff wrote: > >> The fix has been merged into master and will be backported soon. > > Amazing, thanks! > &g

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

> You set mds_beacon_grace ? Yes, as I said. It seemed to have no effect or at least none that I could see. The kick timeout seemed random after all. I even set it to something ridiculous like 1800 and the MDS were still timed out. Sometimes they got to 20M inodes, sometimes only to a few 100k.

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

Hey Patrick, I just wanted to give you some feedback about how 14.2.5 is working for me. I've had the chance to test it for a day now and overall, the experience is much better, although not perfect (perhaps far from it). I have two active MDS (I figured that'd spread the meta data load a li

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

Have you already tried to adjust the "mds_cache_memory_limit" and or "ceph tell mds.* cache drop"? I really wonder how the MDS copes with that with milions of CAPS. I played with the cache size, yeah. I kind of need a large cache, otherwise everything is just slow and I'm constantly getting cac

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

Hi, my MDS failed again, but this time I cannot recover it by deleting the mds*_openfiles .0 object. The startup behaviour is also different. Both inode count and cache size stay at zero while the MDS is replaying. When I set the MDS log level to 7, I get tons of these messages: 2020-01-06 11:59:

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

lowed "self-healing", but I still consider this a severe bug. On 06/01/2020 12:05, Janek Bevendorff wrote: > Hi, my MDS failed again, but this time I cannot recover it by deleting > the mds*_openfiles .0 object. The startup behaviour is also different. > Both inode count and cache

[ceph-users] Re: [Ceph-users] Re: MDS failing under load with large cache sizes

l recovery, "ceph fs status" would still claim that all of then were connected, which was abviously not true.On 7 Jan 2020 2:43 pm, Stefan Kooman wrote:Quoting Janek Bevendorff (janek.bevendo...@uni-weimar.de): > Update: turns out I just had to wait for an hour. The MDSs were sending >

[ceph-users] Provide more documentation for MDS performance tuning on large file systems

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache

[ceph-users] Re: Ceph MDS randomly hangs with no useful error message

The latest one. 14.2.6.On 26 Jan 2020 4:23 pm, "Yan, Zheng" wrote:On Wed, Jan 22, 2020 at 6:22 PM Janek Bevendorff wrote: > > > > I don't find any clue from the backtrace. please run 'ceph daemon > > mds. dump_historic_ops' and ''cep

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.6

I can report similar results, although it's probably not just due to cluster size. Our cluster has 1248 OSDs at the moment and we have three active MDSs to spread the metadata operations evenly. However, I noticed that it isn't spread evenly at all. Usually, it's just one MDS (in our case mds.

[ceph-users] Re: Unexpected recovering after nautilus 14.2.7 -> 14.2.8

I also had some inadvertent recovery going on, although I think it started after I had restarted all MON, MGR, and MDS nodes and before I started restarting OSDs. On 05/03/2020 09:49, Dan van der Ster wrote: Did you have `144 total, 144 up, 144 in` also before the upgrade? If an osd was out,

[ceph-users] MGRs failing once per day and generally slow response times

Hi, I've always had some MGR stability issues with daemons crashing at random times, but since the upgrade to 14.2.8 they regularly stop responding after some time until I restart them (which I have to do at least once a day). I noticed right after the upgrade that the prometheus module was

  1   2   >