Hi,
First, sorry for the typo, the first line should be " I configure a
first cephfs setup using k=6 and m=2 ..."
Second I manage to solve the problem (with chatGPT 🙂). Indeed cephadm
do many things "automagically" when deploying, but undoing things is
less straightforward! Many steps should be done by hand. The additional
missing steps were:
- removing mds (they were still active)
- removing the client (still registred)
and... reboot the entire cluster to be sure that all is starting fine!
May be there is a tutorial detailing how to undo the things in the large
Ceph documentation ?
Now my new setup is running.
Next step is to crash a node and check that Ceph allow read/write with
my setup while redistributing the chunks between the 4 nodes remaining.
Patrick
Le 15/07/2025 à 09:35, Patrick Begou a écrit :
Hi all,
I'm deploying Squid.
I configure a first cephfs setup using k=6 and m=8 but this was a bad
idea for my 5 nodes cluster. So I try to roll back and build a k=5 and
m=3.
But it remains some occurrences of my old setup when allowing cephfs
clients and I do not understand how to remove them. My previous status
was:
# ceph osd pool ls
.mgr
ec-62-pool
cephfs.data-ec62-vol.meta
cephfs.data-ec62-vol.data
to remove it I ran:
# ceph fs fail data-ec62-vol
# ceph fs rm data-ec62-vol --yes-i-really-mean-it
# ceph config set mon mon_allow_pool_delete true
# ceph osd pool delete ec-62-pool ec-62-pool
--yes-i-really-really-mean-it
# ceph osd pool delete cephfs.data-ec62-vol.data
cephfs.data-ec62-vol.data --yes-i-really-really-mean-it
# ceph osd pool delete cephfs.data-ec62-vol.meta
cephfs.data-ec62-vol.meta --yes-i-really-really-mean-it
# ceph config set mon mon_allow_pool_delete false
# ceph osd pool ls
   .mgr
I then remove my erasure code profile:
# ceph osd erasure-code-profile ls
   default
   ec-62-profile-isa
# ceph osd erasure-code-profile rm ec-62-profile-isa
# ceph osd erasure-code-profile ls
   default
And the corresponding crush rule:
# ceph osd crush rule ls
   replicated_rule
   ec-62-pool
# ceph osd crush rule rm ec-62-pool
Then I create my new config:
# ceph osd erasure-code-profile set ec-53-profile-isa k=5 m=3
crush-failure-domain=host plugin=isa
# ceph osd erasure-code-profile ls
   default
   ec-53-profile-isa
# ceph osd pool create ec-53-pool erasure ec-53-profile-isa
   pool 'ec-53-pool' created
# ceph osd pool set ec-53-pool allow_ec_overwrites true
   set pool 5 allow_ec_overwrites to true
# ceph fs volume create data
# ceph fs add_data_pool data ec-53-pool
Last I create a folder in this fs and set it to the erasure coding pool:
# ceph-fuse /mnt
# cd /mnt
# mkdir data
# setfattr -n ceph.dir.layout.pool -v ec-53-pool /mnt/data
But when I want to create the new Key for the clients with:
# ceph fs authorize data client.whitaker / r root_squash /data rw
This still rely on the old data-ec62-vol setup:
[client.whitaker]
   key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
   caps mds = "allow r fsname=data-ec62-vol, allow rw
fsname=data-ec62-vol path=data, allow r fsname=data root_squash, allow
rw fsname=data path=data"
   caps mon = "allow r fsname=data-ec62-vol, allow r fsname=data"
   caps osd = "allow rw tag cephfs data=data-ec62-vol, allow rw tag
cephfs data=data"
And "ceph auth ls" show several remaining occurrences for the old
data-ec62-vol setup:
mds.data-ec62-vol.whitaker02-ceph.jbqyuh
       key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
       caps: [mds] allow
       caps: [mon] profile mds
       caps: [osd] allow rw tag cephfs *=*
mds.data-ec62-vol.whitaker03-ceph.qtaqtn
       key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
       caps: [mds] allow
       caps: [mon] profile mds
       caps: [osd] allow rw tag cephfs *=*
client.whitaker
       key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
       caps: [mds] allow r fsname=data-ec62-vol, allow rw
fsname=data-ec62-vol path=data, allow r fsname=data root_squash, allow
rw fsname=data path=data
       caps: [mon] allow r fsname=data-ec62-vol, allow r fsname=data
       caps: [osd] allow rw tag cephfs data=data-ec62-vol, allow rw
tag cephfs data=data
How can I build a correct setup for these keys ?
Thank
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
