----- Le 9 Mai 25, à 14:10, Frédéric Nass <frederic.n...@univ-lorraine.fr> a
écrit :
> Hi Edouard,
> ----- Le 8 Mai 25, à 10:15, Edouard FAZENDA <e.faze...@csti.ch> a écrit :
>> Dear all,
>> I have the following issue on my Ceph cluster MDSs behind on trimming on my
>> Ceph
>> Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2.
>> Here some cluster logs :
>> 8/5/25 09:00 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
>> 8/5/25 08:50 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
>> 8/5/25 08:40 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming
>> (326/128) max_segments: 128, num_segments: 326
>> 8/5/25 08:40 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming
>> (326/128) max_segments: 128, num_segments: 326
>> 8/5/25 08:40 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
>> 8/5/25 08:40 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
>> 8/5/25 08:33 AM [WRN] Health check update: 2 MDSs behind on trimming
>> (MDS_TRIM)
>> 8/5/25 08:33 AM [WRN] Health check failed: 1 MDSs behind on trimming
>> (MDS_TRIM)
>> 8/5/25 08:30 AM [INF] overall HEALTH_OK
>> 8/5/25 08:22 AM [INF] Cluster is now healthy
>> 8/5/25 08:22 AM [INF] Health check cleared: MDS_TRIM (was: 1 MDSs behind on
>> trimming)
>> 8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming
>> (525/128)
>> 8/5/25 08:22 AM [WRN] Health check update: 1 MDSs behind on trimming
>> (MDS_TRIM)
>> 8/5/25 08:22 AM [INF] MDS health message cleared (mds.?): Behind on trimming
>> (525/128)
>> 8/5/25 08:20 AM [WRN] overall HEALTH_WARN 2 MDSs behind on trimming
>> 8/5/25 08:10 AM [WRN] mds.cephfs.node2.isqjza(mds.0): Behind on trimming
>> (332/128) max_segments: 128, num_segments: 332
>> 8/5/25 08:10 AM [WRN] mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming
>> (332/128) max_segments: 128, num_segments: 332
>> 8/5/25 08:10 AM [WRN] [WRN] MDS_TRIM: 2 MDSs behind on trimming
>> 8/5/25 08:10 AM [WRN] Health detail: HEALTH_WARN 2 MDSs behind on trimming
>> 8/5/25 08:03 AM [WRN] Health check update: 2 MDSs behind on trimming
>> (MDS_TRIM)
>> 8/5/25 08:03 AM [WRN] Health check failed: 1 MDSs behind on trimming
>> (MDS_TRIM)
>> 8/5/25 08:00 AM [INF] overall HEALTH_OK
>> #ceph fs status
>> cephfs - 50 clients
>> ======
>> RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
>> 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818
>> 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0
>> POOL TYPE USED AVAIL
>> cephfs_metadata metadata 2630M 2413G
>> cephfs_data data 12.7T 3620G
>> STANDBY MDS
>> cephfs.node3.vdicdn
>> MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8)
>> squid (stable)
>> # ceph versions
>> {
>> "mon": {
>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid
>> (stable)":
>> 3
>> },
>> "mgr": {
>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid
>> (stable)":
>> 2
>> },
>> "osd": {
>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid
>> (stable)":
>> 18
>> },
>> "mds": {
>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid
>> (stable)":
>> 3
>> },
>> "rgw": {
>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid
>> (stable)":
>> 6
>> },
>> "overall": {
>> "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid
>> (stable)":
>> 32
>> }
>> }
>> #ceph orch ps --daemon-type mds
>> NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID
>> CONTAINER
>> ID
>> mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2
>> 4892a7ef541b 8dd8db30a1de
>> mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2
>> 4892a7ef541b 7b9d5b692764
>> mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2
>> 4892a7ef541b d2de22a15e18
>> root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn |
>> egrep
>> "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
>> mds_cache_memory_limit 4294967296 default
>> mds_cache_trim_decay_rate 1.000000 default
>> mds_cache_trim_threshold 262144 default
>> mds_recall_max_caps 30000 default
>> mds_recall_max_decay_rate 1.500000 default
>> root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza |
>> egrep
>> "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
>> mds_cache_memory_limit 4294967296 default
>> mds_cache_trim_decay_rate 1.000000 default
>> mds_cache_trim_threshold 262144 default
>> mds_recall_max_caps 30000 default
>> mds_recall_max_decay_rate 1.500000 default
>> root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk |
>> egrep
>> "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate"
>> mds_cache_memory_limit 4294967296 default
>> mds_cache_trim_decay_rate 1.000000 default
>> mds_cache_trim_threshold 262144 default
>> mds_recall_max_caps 30000 default
>> mds_recall_max_decay_rate 1.500000 default
>> # ceph mds stat
>> cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby
>> Do you have an idea on what could happen ? Should I increate
>> mds_cache_trim_decay_rate ?
>> I saw the folloing issue : [ https://tracker.ceph.com/issues/66948 | Bug
>> #66948:
>> mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on
>> trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ] ( [
>> https://github.com/ceph/ceph/pull/60838 | squid: mds: trim mdlog when
>> segments
>> exceed threshold and trim was idle by vshankar · Pull Request #60838 ·
>> ceph/ceph · GitHub ] ) maybe related ?
> There's a fair chance, yes as you said MDS_TRIM alert came after the upgrade,
> Reef is immune to this bug (as not using major/minor log segment changes) and
> Squid v19.2.2 does not contain the fix.
> Wou could try decreasing mds_recall_max_decay_rate to 1 (instead of 1.5)
> Regards,
> Frédéric.
Apologies, the email was sent prematurely...
What you could try waiting for the fix is to:
- increase mds_log_max_segments to 256 (defaults to 128)
And eventually:
- reduce mds_recall_max_decay_rate to 1 (defaults to 1.5)
- reduce mds_recall_max_decay_threshold to 32K (defaults to 128K)
- increase mds_recall_global_max_decay_threshold to 256K (defaults to 128K)
to allow the MDS to reclaim client caps in a more aggressive manner, though I'm
not sure this will prevent MDS_TRIM alert from being triggered.
Regards,
Frédéric.
>> Thanks for the help 😊
>> Best Regards, Edouard Fazenda.
>> [ https://www.csti.ch/ ]
>> S wiss C loud P rovider
>> Edouard Fazenda
>> Technical Support
>> [ https://www.csti.ch/ ]
>> Chemin du Curé-Desclouds, 2
>> CH-1226 Thonex
>> +41 22 869 04 40
>> [ https://mail.univ-lorraine.fr/www.csti.ch | www.csti.ch ]
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
