Hi, we knew that only newly created data would belong to the new pool. Why is it a problem though? Is it possible to declare all data in the /temp directory corrupt, allow me to delete it, and move on with life?
Thank you, --Max ________________________________ From: Eugen Block <ebl...@nde.ag> Sent: Tuesday, May 6, 2025 2:53 AM To: Maxim Goncharov <m...@mit.edu> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Need help with MDS stuck in readonly state Hi, it seems like you only changed the xattr, but that doesn't really move the objects into the new pool, only new objects would be created in the additional pool. There was a thread [0] on this list a few months ago, pointing to [1] how to actually migrate data to a different pool. I'm just not sure how to get out of this situation. Based on the description in the docs [2], writes to the metadata pool are blocked. So one option could be to reset the journal (take a backup first) as described in the troubleshooting guide [3]. Maybe someone else has other ideas... Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/M72ZAQPMX3SPCDZFT7XLG5IOQHJJPUOA/ [1] https://ewal.dev/cephfs-migrating-files-between-pools [2] https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-health-read-only [3] https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#journal-truncation Zitat von Maxim Goncharov <m...@mit.edu>: > Hi, here is the sequence of commands I executed to change the > directory to a different pools, and then exact sequence in reverse. > > system("ceph osd crush rule create-replicated replicated_hdd_rule > default host hdd"); > system("ceph osd pool create tier2mit_rep_hdd replicated --bulk"); > system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule"); > system("ceph osd pool set tier2mit_rep_hdd size 2"); > system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd"); > system("cephfs-shell setxattr /rootfolder/cms/store/temp/user > ceph.dir.layout.pool tier2mit_rep_hdd"); > > cephfs-shell setxattr /rootfolder/cms/store/temp/user > ceph.dir.layout.pool tier2mit_ec_hdd > then removed pool from fs, and deleted it. > > One of the things we tried was to put pool in failed state and then > rejoin it (ceph fs fail tier2mit; ceph fs set tier2mit joinable > true), here is a log from MDS, you can see where it puts itself in > readonly mode. > > https://www.cmsaf.mit.edu/various/messages > > Thankyou, --Max > > > > > > > > ________________________________ > From: Eugen Block <ebl...@nde.ag> > Sent: Monday, May 5, 2025 5:47 AM > To: ceph-users@ceph.io <ceph-users@ceph.io> > Subject: [ceph-users] Re: Need help with MDS stuck in readonly state > > Hi, > > do you have logs from the MDS? And how exactly did you move the /temp > directory to a different pool? That might help understand what > happened. There was a thread here a couple of weeks ago how about > moving contents to a different CephFS pool. > > Zitat von Maxim Goncharov <m...@mit.edu>: > >> Hi Everyone, >> I have a CephFS cluster that has MDS stuck in read only mode, and I >> do not know what can be done to return the cluster into the writable >> mode: >> health: HEALTH_WARN >> 1 MDSs are read only >> 1 MDSs behind on trimming >> How I got there: we have an FS where we store data in EC pool. We >> have one subdirectory that contains a lot of transitory data >> (constant write/read/delete). I wanted to put this subdirectory >> (let's call it /temp) into a separate replicated pool. >> I did: created replicated pool, added it to the FS, switched /temp >> to that replicated pool. >> All worked for about 24 hours and then became obvious it was not a >> good idea, so I reversed all steps: switched /temp to the original >> EC pool, removed replicated pool from FS, deleted the replicated pool. >> That is when MDS went into read only state. I can see inodes that >> were created in /temp when it belonged to the replicated pool. >> >> I tried many different things that I found in documentation, nothing >> works. What should I do in order for the FS become writable again? >> >> Thank you, --Max >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
