Hi Edouard, Sorry to hear that, although I'm not that surprised. I think you'll have to wait for the fix.
Regards, Frédéric. ----- Le 13 Mai 25, à 15:41, Edouard FAZENDA <e.faze...@csti.ch> a écrit : > Dear Frederic, > I have applied the settings you have provided, unfortunately the cluster was > back green and yellow afterward with still MDS on trimming again. > Thanks for the help > Best Regards, > [ https://www.csti.ch/ ] > S wiss C loud P rovider > Edouard Fazenda > Technical Support > [ https://www.csti.ch/ ] > Chemin du Curé-Desclouds, 2 > CH-1226 Thonex > +41 22 869 04 40 > [ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ] > From: Frédéric Nass <frederic.n...@univ-lorraine.fr> > Sent: vendredi, 9 mai 2025 14:20 > To: Edouard FAZENDA <e.faze...@csti.ch> > Cc: ceph-users <ceph-users@ceph.io>; Kelian SAINT-BONNET > <k.saintbon...@csti.ch> > Subject: Re: [ceph-users] 2 MDSs behind on trimming on my Ceph Cluster since > the > upgrade from 18.2.6 (reef) to 19.2.2 (squid) > ----- Le 9 Mai 25, à 14:10, Frédéric Nass < [ > mailto:frederic.n...@univ-lorraine.fr | frederic.n...@univ-lorraine.fr ] > a > écrit : >> Hi Edouard, >> ----- Le 8 Mai 25, à 10:15, Edouard FAZENDA < [ mailto:e.faze...@csti.ch | >> e.faze...@csti.ch ] > a écrit : >>> Dear all, >>> I have the following issue on my Ceph cluster MDSs behind on trimming on my >>> Ceph >>> Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2. >>> Here some cluster logs : >>> 8/5/25 09:00 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming >>> 8/5/25 08:50 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming >>> 8/5/25 08:40 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming >>> (326/128) max_segments: 128, num_segments: 326 >>> 8/5/25 08:40 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming >>> (326/128) max_segments: 128, num_segments: 326 >>> 8/5/25 08:40 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming >>> 8/5/25 08:40 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming >>> 8/5/25 08:33 AM [WRN] Health check update: 2 MDSs behind on trimming >>> (MDS_TRIM) >>> 8/5/25 08:33 AM [WRN] Health check failed: 1 MDSs behind on trimming >>> (MDS_TRIM) >>> 8/5/25 08:30 AM [INF] overall HEALTH_OK >>> 8/5/25 08:22 AM [INF] Cluster is now healthy >>> 8/5/25 08:22 AM [INF] Health check cleared: MDS_TRIM (was: 1 MDSs behind on >>> trimming) >>> 8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming >>> (525/128) >>> 8/5/25 08:22 AM [WRN] Health check update: 1 MDSs behind on trimming >>> (MDS_TRIM) >>> 8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming >>> (525/128) >>> 8/5/25 08:20 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming >>> 8/5/25 08:10 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming >>> (332/128) max_segments: 128, num_segments: 332 >>> 8/5/25 08:10 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming >>> (332/128) max_segments: 128, num_segments: 332 >>> 8/5/25 08:10 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming >>> 8/5/25 08:10 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming >>> 8/5/25 08:03 AM [WRN] Health check update: 2 MDSs behind on trimming >>> (MDS_TRIM) >>> 8/5/25 08:03 AM [WRN] Health check failed: 1 MDSs behind on trimming >>> (MDS_TRIM) >>> 8/5/25 08:00 AM [INF] overall HEALTH_OK >>> #ceph fs status >>> cephfs - 50 clients >>> ====== >>> RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS >>> 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818 >>> 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0 >>> POOL TYPE USED AVAIL >>> cephfs_metadata metadata 2630M 2413G >>> cephfs_data data 12.7T 3620G >>> STANDBY MDS >>> cephfs.node3.vdicdn >>> MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) >>> squid (stable) >>> # ceph versions >>> { >>> "mon": { >>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid >>> (stable)": >>> 3 >>> }, >>> "mgr": { >>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid >>> (stable)": >>> 2 >>> }, >>> "osd": { >>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid >>> (stable)": >>> 18 >>> }, >>> "mds": { >>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid >>> (stable)": >>> 3 >>> }, >>> "rgw": { >>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid >>> (stable)": >>> 6 >>> }, >>> "overall": { >>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid >>> (stable)": >>> 32 >>> } >>> } >>> #ceph orch ps --daemon-type mds >>> NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID >>> CONTAINER >>> ID >>> mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 >>> 4892a7ef541b 8dd8db30a1de >>> mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 >>> 4892a7ef541b 7b9d5b692764 >>> mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 >>> 4892a7ef541b d2de22a15e18 >>> root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | >>> egrep >>> "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" >>> mds_cache_memory_limit 4294967296 default >>> mds_cache_trim_decay_rate 1.000000 default >>> mds_cache_trim_threshold 262144 default >>> mds_recall_max_caps 30000 default >>> mds_recall_max_decay_rate 1.500000 default >>> root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | >>> egrep >>> "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" >>> mds_cache_memory_limit 4294967296 default >>> mds_cache_trim_decay_rate 1.000000 default >>> mds_cache_trim_threshold 262144 default >>> mds_recall_max_caps 30000 default >>> mds_recall_max_decay_rate 1.500000 default >>> root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | >>> egrep >>> "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" >>> mds_cache_memory_limit 4294967296 default >>> mds_cache_trim_decay_rate 1.000000 default >>> mds_cache_trim_threshold 262144 default >>> mds_recall_max_caps 30000 default >>> mds_recall_max_decay_rate 1.500000 default >>> # ceph mds stat >>> cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby >>> Do you have an idea on what could happen ? Should I increate >>> mds_cache_trim_decay_rate ? >>> I saw the folloing issue : [ https://tracker.ceph.com/issues/66948 | Bug >>> #66948: >>> mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on >>> trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ] ( [ >>> https://github.com/ceph/ceph/pull/60838 | squid: mds: trim mdlog when >>> segments >>> exceed threshold and trim was idle by vshankar · Pull Request #60838 · >>> ceph/ceph · GitHub ] ) maybe related ? >> There's a fair chance, yes as you said MDS_TRIM alert came after the upgrade, >> Reef is immune to this bug (as not using major/minor log segment changes) and >> Squid v19.2.2 does not contain the fix. >> Wou could try decreasing mds_recall_max_decay_rate to 1 (instead of 1.5) >> Regards, >> Frédéric. > Apologies, the email was sent prematurely... > What you could try waiting for the fix is to: > - increase mds_log_max_segments to 256 (defaults to 128) > And eventually: > - reduce mds_recall_max_decay_rate to 1 (defaults to 1.5) > - reduce mds_recall_max_decay_threshold to 32K (defaults to 128K) > - increase mds_recall_global_max_decay_threshold to 256K (defaults to 128K) > to allow the MDS to reclaim client caps in a more aggressive manner, though > I'm > not sure this will prevent MDS_TRIM alert from being triggered. > Regards, > Frédéric. >>> Thanks for the help 😊 >>> Best Regards, Edouard Fazenda. >>> [ https://www.csti.ch/ ] >>> S wiss C loud P rovider >>> Edouard Fazenda >>> Technical Support >>> [ https://www.csti.ch/ ] >>> Chemin du Curé-Desclouds, 2 >>> CH-1226 Thonex >>> +41 22 869 04 40 >>> [ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ] >>> _______________________________________________ >>> ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io >>> ] >>> To unsubscribe send an email to [ mailto:ceph-users-le...@ceph.io | >>> ceph-users-le...@ceph.io ] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
