Hi, here is the sequence of commands I executed to change the directory to a
different pools, and then exact sequence in reverse.
system("ceph osd crush rule create-replicated replicated_hdd_rule default host
hdd");
system("ceph osd pool create tier2mit_rep_hdd replicated --bulk");
system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule");
system("ceph osd pool set tier2mit_rep_hdd size 2");
system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd");
system("cephfs-shell setxattr /rootfolder/cms/store/temp/user
ceph.dir.layout.pool tier2mit_rep_hdd");
cephfs-shell setxattr /rootfolder/cms/store/temp/user ceph.dir.layout.pool
tier2mit_ec_hdd
then removed pool from fs, and deleted it.
One of the things we tried was to put pool in failed state and then rejoin it
(ceph fs fail tier2mit; ceph fs set tier2mit joinable true), here is a log from
MDS, you can see where it puts itself in readonly mode.
https://www.cmsaf.mit.edu/various/messages
Thankyou, --Max
________________________________
From: Eugen Block <ebl...@nde.ag>
Sent: Monday, May 5, 2025 5:47 AM
To: ceph-users@ceph.io <ceph-users@ceph.io>
Subject: [ceph-users] Re: Need help with MDS stuck in readonly state
Hi,
do you have logs from the MDS? And how exactly did you move the /temp
directory to a different pool? That might help understand what
happened. There was a thread here a couple of weeks ago how about
moving contents to a different CephFS pool.
Zitat von Maxim Goncharov <m...@mit.edu>:
> Hi Everyone,
> I have a CephFS cluster that has MDS stuck in read only mode, and I
> do not know what can be done to return the cluster into the writable
> mode:
> health: HEALTH_WARN
> 1 MDSs are read only
> 1 MDSs behind on trimming
> How I got there: we have an FS where we store data in EC pool. We
> have one subdirectory that contains a lot of transitory data
> (constant write/read/delete). I wanted to put this subdirectory
> (let's call it /temp) into a separate replicated pool.
> I did: created replicated pool, added it to the FS, switched /temp
> to that replicated pool.
> All worked for about 24 hours and then became obvious it was not a
> good idea, so I reversed all steps: switched /temp to the original
> EC pool, removed replicated pool from FS, deleted the replicated pool.
> That is when MDS went into read only state. I can see inodes that
> were created in /temp when it belonged to the replicated pool.
>
> I tried many different things that I found in documentation, nothing
> works. What should I do in order for the FS become writable again?
>
> Thank you, --Max
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
