Hi,
it seems like you only changed the xattr, but that doesn't really move
the objects into the new pool, only new objects would be created in
the additional pool. There was a thread [0] on this list a few months
ago, pointing to [1] how to actually migrate data to a different pool.
I'm just not sure how to get out of this situation. Based on the
description in the docs [2], writes to the metadata pool are blocked.
So one option could be to reset the journal (take a backup first) as
described in the troubleshooting guide [3]. Maybe someone else has
other ideas...
Regards,
Eugen
[0]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/M72ZAQPMX3SPCDZFT7XLG5IOQHJJPUOA/
[1] https://ewal.dev/cephfs-migrating-files-between-pools
[2]
https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-health-read-only
[3]
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#journal-truncation
Zitat von Maxim Goncharov <m...@mit.edu>:
Hi, here is the sequence of commands I executed to change the
directory to a different pools, and then exact sequence in reverse.
system("ceph osd crush rule create-replicated replicated_hdd_rule
default host hdd");
system("ceph osd pool create tier2mit_rep_hdd replicated --bulk");
system("ceph osd pool set tier2mit_rep_hdd crush_rule replicated_hdd_rule");
system("ceph osd pool set tier2mit_rep_hdd size 2");
system("ceph fs add_data_pool tier2mit tier2mit_rep_hdd");
system("cephfs-shell setxattr /rootfolder/cms/store/temp/user
ceph.dir.layout.pool tier2mit_rep_hdd");
cephfs-shell setxattr /rootfolder/cms/store/temp/user
ceph.dir.layout.pool tier2mit_ec_hdd
then removed pool from fs, and deleted it.
One of the things we tried was to put pool in failed state and then
rejoin it (ceph fs fail tier2mit; ceph fs set tier2mit joinable
true), here is a log from MDS, you can see where it puts itself in
readonly mode.
https://www.cmsaf.mit.edu/various/messages
Thankyou, --Max
________________________________
From: Eugen Block <ebl...@nde.ag>
Sent: Monday, May 5, 2025 5:47 AM
To: ceph-users@ceph.io <ceph-users@ceph.io>
Subject: [ceph-users] Re: Need help with MDS stuck in readonly state
Hi,
do you have logs from the MDS? And how exactly did you move the /temp
directory to a different pool? That might help understand what
happened. There was a thread here a couple of weeks ago how about
moving contents to a different CephFS pool.
Zitat von Maxim Goncharov <m...@mit.edu>:
Hi Everyone,
I have a CephFS cluster that has MDS stuck in read only mode, and I
do not know what can be done to return the cluster into the writable
mode:
health: HEALTH_WARN
1 MDSs are read only
1 MDSs behind on trimming
How I got there: we have an FS where we store data in EC pool. We
have one subdirectory that contains a lot of transitory data
(constant write/read/delete). I wanted to put this subdirectory
(let's call it /temp) into a separate replicated pool.
I did: created replicated pool, added it to the FS, switched /temp
to that replicated pool.
All worked for about 24 hours and then became obvious it was not a
good idea, so I reversed all steps: switched /temp to the original
EC pool, removed replicated pool from FS, deleted the replicated pool.
That is when MDS went into read only state. I can see inodes that
were created in /temp when it belonged to the replicated pool.
I tried many different things that I found in documentation, nothing
works. What should I do in order for the FS become writable again?
Thank you, --Max
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
