----- Le 7 Mai 25, à 15:45, Maxim Goncharov m...@mit.edu a écrit : > Hi Frederic,
> Thank you for looking into it more. We indeed tried the steps that you > describe. > The problem was that the MDS would turn readonly before deletion of that /temp > directory could run. Preventing access to the filesystem (ceph fs set cephfs refuse_client_session true) may have helped with that, I believe. > Now we can't try anything else because yesterday we wiped everything out and > bootstrapped a new cluster. > Please please put somewhere in documentation in big red letters not to do what > we have done so that other people would not run into the same issue. Well, this issue was discussed [1], tracked [2] and fixed [3] 8 years ago in Jewel. I don't know why this could happen again as the fix made back then is still present in the code of latest Ceph releases. Something the devs might want to investigate. Given this is a bug that's been fixed in Reef and further releases, it would seem odd to add it to the documentation, or maybe to Quincy documentation? Regards, Frédéric. [1] https://ceph-users.ceph.narkive.com/h9GQKIe4/mds-read-only-state-in-production-cephfs [2] https://tracker.ceph.com/issues/19401 [3] https://github.com/ceph/ceph/pull/14682 > Thank you, --Max > From: Frédéric Nass <frederic.n...@univ-lorraine.fr> > Sent: Wednesday, May 7, 2025 9:36 AM > To: Maxim Goncharov <m...@mit.edu> > Cc: Milan Kupcevic <milan_kupce...@harvard.edu>; ceph-users > <ceph-users@ceph.io> > Subject: Re: Need help with MDS stuck in readonly state > Maxim, > I reproduced this issue on a v19.2.1 lab and I think I find an easy way out. > Here below is what worked on my side (tested multiple times): > - unmount de filesystem from any clients accessing /temp directory > - restart active MDS(s) > - filesystem should now be readwrite (until next flush of the journal) > - umount and remount the fs on an admin node > - remove any files/subdirs in the "temp" tree that's not accessible (or all of > them if you can afford it) using the linux 'rm' command. If the filesystem > already switched to readonly mode due to client activity, do it anyway. > - flush the journal (ceph tell mds.0 flush journal). > - filesystem should stay readwrite. If it switches to readonly again, then try > restarting MDSs and flush again. If it still goes to readonly mode then check > MDS logs for any files left over and repeat this procedure. > Let me know how it goes. > Best, > Frédéric. > ----- Le 7 Mai 25, à 12:29, Maxim Goncharov <m...@mit.edu> a écrit : >> Hi Frederic, >> We are running 19.2.2 squid (stable), all fixes should have been already in, >> I >> hope. >> I do not really understand what commands you are advising us to issue, is >> there >> web page >> that lists a sequence of commands we should run, or please put them in >> e-mail? >> Thank you, --Max >> From: Frédéric Nass <frederic.n...@univ-lorraine.fr> >> Sent: Wednesday, May 7, 2025 6:04 AM >> To: Milan Kupcevic <milan_kupce...@harvard.edu> >> Cc: ceph-users <ceph-users@ceph.io> >> Subject: [ceph-users] Re: Need help with MDS stuck in readonly state >> Hi Milan, >> Great catch! The fix is in Reef v18.2.5 and has been merged to Squid branch, >> so >> it should be in v19.2.3, I think. No fix for EOL Quincy. >> @Maxim, there might be a way to recover from this situation (without going >> through the disaster recovery procedure) by fixing the backtraces or removing >> the /temp directory as a whole - if the MDS allows to do so before switching >> to >> readonly mode - and flushing the journal, but this is not something I tested. >> Your best move from now would be to upgrade to v18.2.7 when it's out. Note >> that >> upgrading to v18.2.5 or v18.2.6 has been highly discouraged recently due to >> bluestore issues. >> Regards, >> Frédéric. >> ----- Le 6 Mai 25, à 19:51, Milan Kupcevic milan_kupce...@harvard.edu a >> écrit : >> > Hi Max, >> > This could be related to the reported issue #63259. See: >> > [ https://tracker.ceph.com/issues/63259#note-11 | >> https://tracker.ceph.com/issues/63259#note-11 ] >> > Milan >> > -- >> > Milan Kupcevic >> > Research Computing Lead Storage Engineer >> > Harvard University >> > HUIT, University Research Computing >> > On 5/1/25 08:22, Maxim Goncharov wrote: >> >> Hi Everyone, >> >> I have a CephFS cluster that has MDS stuck in read only mode, and I do >> >> not know >> >> what can be done to return the cluster into the writable mode: >> >> health: HEALTH_WARN >> >> 1 MDSs are read only >> >> 1 MDSs behind on trimming >> >> How I got there: we have an FS where we store data in EC pool. We have one >> >> subdirectory that contains a lot of transitory data (constant >> >> write/read/delete). I wanted to put this subdirectory (let's call it >> >> /temp) >> >> into a separate replicated pool. >> >> I did: created replicated pool, added it to the FS, switched /temp to that >> >> replicated pool. >> >> All worked for about 24 hours and then became obvious it was not a good >> >> idea, so >> >> I reversed all steps: switched /temp to the original EC pool, removed >> >> replicated pool from FS, deleted the replicated pool. >> >> That is when MDS went into read only state. I can see inodes that were >> >> created >> >> in /temp when it belonged to the replicated pool. >> >> I tried many different things that I found in documentation, nothing >> >> works. What >> >> should I do in order for the FS become writable again? >> >> Thank you, --Max >> >> _______________________________________________ >> >> ceph-users mailing list --ceph-users@ceph.io >> >> To unsubscribe send an email toceph-users-le...@ceph.io >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@ceph.io >> > To unsubscribe send an email to ceph-users-le...@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
