Hi Kasper, Glad to hear you got it fixed. If you want to dig deeper into the root cause, check the mds logs for any backtraces. Here's an issue that comes to mind as possibly causing this for you: https://tracker.ceph.com/issues/64717 https://tracker.ceph.com/issues/64348 https://tracker.ceph.com/issues/68374
Cheers, Dan On Mon, Mar 31, 2025 at 1:37 AM Kasper Rasmussen <kasper_steenga...@hotmail.com> wrote: > > ISSUE FIXED > > After rebooting all clients, and setting - ceph config set mon > mds_beacon_grace 120 > The MDS finally went active > > ________________________________ > From: Kasper Rasmussen <kasper_steenga...@hotmail.com> > Sent: Monday, March 31, 2025 09:46 > To: ceph-users <ceph-users@ceph.io> > Subject: [ceph-users] Ceph MDS stuck in reconnect -> rejoin -> failover loop > > Hi > > Ceph pacific 16.2.15 > > I have 5 MDS hosts, 4 active (4 FS), and 1 standby. > > One MDS was restarted today (as part of OS Patching), resulting in a > failover. This is usually not an issue but today,it got stuck in a reconnect > -> rejoin -> failover loop for the specific FS. > > A ceph fs status shows that during the time the FS is in state "rejoin" the > INOS rise to +50M (usually it is around 10-12M ) > > The memory on the MDS host is eaten, (MDS cache size is 36GB, but it rises to > +140 GB.) > > Finaly it fails over, and the cycle starts over. > > We are currently restarting all clients, in an effort to rule out buggy > clients. > > > Any help on this issue will be very much appreciated. Thank you > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanders...@clyso.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
