[ceph-users] Mixed cluster with AMD64 and ARM64 possible?
Hi, just a quick question: Is a mixed cluster with nodes running AMD64 and ARM64 CPUs possible? Is the cephadm orchestrator able to manage such a cluster? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Mixed cluster with AMD64 and ARM64 possible?
Hi, we have been running such mixed cluster for almost 2 years. It works without any cephadm related issues. (Even during upgrade form ubuntu 20.04 to 22.04 mixing nodes with docker and podman caused no issues.) Just be careful when upgrading, ceph arm64 containers for 18.2.4 are completely broken. ceph-osd in that containers segvfault and only fix is to upgrade to 19.2.0/19.2.1. https://tracker.ceph.com/issues/67213 Adam W dniu 6.03.2025 o 08:32, Robert Sander pisze: Hi, just a quick question: Is a mixed cluster with nodes running AMD64 and ARM64 CPUs possible? Is the cephadm orchestrator able to manage such a cluster? Regards smime.p7s Description: Kryptograficzna sygnatura S/MIME ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] cephfs healthy but mounting it some data cannot be accessed
Hi, we have access and we can mount cephfs volume but for some reason data in it can be listes but when we try to copy some files from it , some can be and some hangs Is there a way to fix it or skip the files we cannot access ? Best regards ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph cluster unable to read/write data properly and cannot recover normally.
Den ons 5 mars 2025 kl 01:59 skrev 小小 <13071007...@163.com>: > Hi, >I'm facing a critical issue with my Ceph cluster. It has become unable to > read/write data properly and cannot recover normally. What steps should I > take to resolve this? Did you do anything to the cluster, or did anything happen to it or the network before this situation we see below? The "6 osds 6 up" makes it look "normal" but have you removed OSDs or a host that made it end up like this? It is very strange if OSDs don't go down and their PGs are unknown, so there is some other information we are lacking here about why so many PGs are not active even if the OSDs are there. > osd: 6 osds: 6 up, 6 in; 169 remapped pgs > pgs: 12.500% pgs unknown > 6.019% pgs not active > 27 unknown > 9 undersized+degraded+peered > 4 undersized+peered > io: > > recovery: 1.3 MiB/s, 20 objects/s， -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute
Hello everyone, I'm seeing some behaviour in CephFS that strikes me as unexpected, and I wonder if others have thoughts about it. Consider this scenario: * Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, CephFS client is running kernel 5.15.0-133-generic. * CephFS is mounted to /mnt, using a CephX identity that has rwp permissions. * CephFS is using a single data pool named cephfs_data. * In the CephFS root directory, there is a subdirectory named "restricted". That subdirectory currently has a single file, "test2.bin", of 8 MiB. There is no other data in the data pool. I can verify this with the following rados command: rados -n client.cephfs -p cephfs_data ls 103.0001 103. Now I run this, from the root of my mounted CephFS: cd /mnt setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted I realise this is naughty, because there is currently a file named /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes on a non-empty directory. However, the command does succeed, and I am able to read back the ceph.dir.layout xattr: getfattr -n ceph.dir.layout restricted # file: restricted ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data pool_namespace=restricted" If at this stage I try to read RADOS objects in the "restricted" namespace, I get an empty list: rados -p cephfs_data -N restricted ls (empty output) Now I move my test file out of the restricted subdirectory and back in: mv /mnt/restricted/test2.bin /mnt/ sync mv /mnt/test2.bin /mnt/restricted/ sync No change: rados -p cephfs_data -N restricted ls (empty output) Next, I try moving the file out, and *copying* it back in: mv /mnt/restricted/test2.bin /mnt/ sync cp /mnt/test2.bin /mnt/restricted/ sync Now, I do see objects in the "restricted" namespace: rados -p cephfs_data -N restricted ls 10001fd.0001 10001fd. Also, if I create a *new* file in the "restricted" subdirectory, then its RADOS objects do end up in the correct namespace: dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M rados -p cephfs_data -N restricted ls 10001fe. 10001fe.0002 10001fe.0001 10001fd.0001 10001fd. By contrast, here's what happens to a non-empty *file,* when I try to set ceph.file.layout: dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin setfattr: test4.bin: Directory not empty So that fails, somewhat expectedly (although the error message is odd). In summary, there are two things that confuse me here: 1. Why does setting ceph.dir.layout.pool_namespace on a non-empty directory succeed, when setting ceph.file.layout.pool_namespace on a non-empty file fails (and even confusingly with a "Directory not empty" message)? 2. Considering that setting ceph.dir.layout.pool_namespace on a non-empty directory does succeed, why does mv'ing a file to a directory with a different pool_namespace, and then mv'ing it back, not result in its RADOS objects moving to the other namespace? So I'm curious: is this a bug (or two), or have I been misunderstanding what's actually the expected behaviour? Cheers, Florian smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Error Removing Zone from Zonegroup in Multisite Setup
Hi everyone, We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation. I ran radosgw-admin zonegroup remove --rgw-zonegroup= --rgw-zone= --rgw-realm= from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup= --rgw-zone= --rgw-realm= However, the period update --commit command returns the error: ERROR: zonegroup does not contain zone id The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors. Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you! Best regards, Mahnoosh ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Module 'devicehealth' has failed
And do you also have the device_health_metrics pool? During one of the upgrades to Quincy or so the older device_health_metrics should have been renamed. But on one customer cluster I found that both were still there, although that didn't cause any trouble. I don't really fully grasp yet what exactly goes wrong here, but maybe you have some log context as well, not only the stack trace? Zitat von Alex from North : yes, I do .mgr 10 1 769 KiB2 2.3 MiB 04.7 PiB ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Error Removing Zone from Zonegroup in Multisite Setup
Hi, I've had similar issues before. radosgw-admin zonegroup/zone tooling is really bad in general. As far as I know you should remove zone before deleting it form zonegroup but I don't think I've ever managed to remove zone cleanly from the cluster. (I would be thankful if someone shared a correct way) (https://www.ibm.com/docs/en/storage-ceph/7?topic=zones-deleting-zone) To clean this up you may have to dump your zonegroup to file, edit json manually and upload it back to ceph without broken zone. If you have issues with deleting the zone itself from the local cluster RGW configuration you may have to hunt down objects containing it's IDs in .rgw.root pool. It's messy and dangerous so, don't forget to make a backup before you start ;) best regards, Adam W dniu 5.03.2025 o 18:02, Mahnoosh Shahidi pisze: Hi everyone, We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation. I ran radosgw-admin zonegroup remove --rgw-zonegroup= --rgw-zone= --rgw-realm= from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup= --rgw-zone= --rgw-realm= However, the period update --commit command returns the error: ERROR: zonegroup does not contain zone id The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors. Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you! Best regards, Mahnoosh ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io smime.p7s Description: Kryptograficzna sygnatura S/MIME ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute
Hi, just some assumptions based on my experience with cephfs: - you cannot change existing objects; setting a different pool will not automagically move data. the data pool seems to be stored in the inode information - this also explains why changing the pool of a file does not work. ceph is unable to move the data on its own - so after changing the pool for a directory, all existing files will remain in their original pool - only newly created files will inherit the changed pool assignment - moving a file does not create a new inode, so its pool assignment cannot be changed by moving The last time we want to perform a large scale pool change we had to copy each file / directory to actually move the data. Best regards, Burkhard Linke ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Error Removing Zone from Zonegroup in Multisite Setup
On Wed, Mar 5, 2025 at 9:34 AM Adam Prycki wrote: > Hi, > > I've had similar issues before. > radosgw-admin zonegroup/zone tooling is really bad in general. > > > As far as I know you should remove zone before deleting it form > zonegroup but I don't think I've ever managed to remove zone cleanly > from the cluster. (I would be thankful if someone shared a correct way) > (https://www.ibm.com/docs/en/storage-ceph/7?topic=zones-deleting-zone) > > To clean this up you may have to dump your zonegroup to file, edit json > manually and upload it back to ceph without broken zone. > > If you have issues with deleting the zone itself from the local cluster > RGW configuration you may have to hunt down objects containing it's IDs > in .rgw.root pool. It's messy and dangerous so, don't forget to make a > backup before you start ;) > > > best regards, > Adam > > > W dniu 5.03.2025 o 18:02, Mahnoosh Shahidi pisze: > > Hi everyone, > > > > We have a Ceph multisite setup with a zonegroup containing three zones: > one > > master and two secondary zones, one of which is designated as an archive. > > After recently upgrading to Ceph 19.2.1, I attempted to remove the > archive > > zone from the zonegroup following the steps in the Ceph documentation. > > > > I ran > > radosgw-admin zonegroup remove --rgw-zonegroup= > > --rgw-zone= --rgw-realm= > > from the cluster containing the archive zone, followed by > > radosgw-admin period update --commit --rgw-zonegroup= > > --rgw-zone= --rgw-realm= > > > > However, the period update --commit command returns the error: > > ERROR: zonegroup does not contain zone id > > The zone appears correctly in the zonegroup before the removal, and I > > expected the archive zone to be successfully removed with the period > update > > committing without errors. > > > > Is there an additional step required, or could this be a bug related to > > Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you! > > > > it does appear to be a regression. have raised a tracker https://tracker.ceph.com/issues/70323 you don't see that zone entry when you run 'radosgw-admin zonegroup get' command on the archive zone anymore, do you? what happens when you run 'radosgw-admin period update' and then 'radosgw-admin period commit' separately? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute
Hi Florian, Point 1 is certainly a bug regarding the choice of terms in the response (confusion between file and directory). Point 2 is known (cf. https://ewal.dev/cephfs-migrating-files-between-pools) and described in the documentation: only new files are written to the new pool after setting the new layout. The 'mv' command does not move the Rados objects associated with the file from one pool to another. Only reading and rewriting the file does. What reflections/impediments led to this? I do not know. But what you observed is expected. Hope this helps, Cheers, Frédéric. De : Florian Haas Envoyé : mercredi 5 mars 2025 14:07 À : ceph-users Objet : [ceph-users] Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute Hello everyone, I'm seeing some behaviour in CephFS that strikes me as unexpected, and I wonder if others have thoughts about it. Consider this scenario: * Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, CephFS client is running kernel 5.15.0-133-generic. * CephFS is mounted to /mnt, using a CephX identity that has rwp permissions. * CephFS is using a single data pool named cephfs_data. * In the CephFS root directory, there is a subdirectory named "restricted". That subdirectory currently has a single file, "test2.bin", of 8 MiB. There is no other data in the data pool. I can verify this with the following rados command: rados -n client.cephfs -p cephfs_data ls 103.0001 103. Now I run this, from the root of my mounted CephFS: cd /mnt setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted I realise this is naughty, because there is currently a file named /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes on a non-empty directory. However, the command does succeed, and I am able to read back the ceph.dir.layout xattr: getfattr -n ceph.dir.layout restricted # file: restricted ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data pool_namespace=restricted" If at this stage I try to read RADOS objects in the "restricted" namespace, I get an empty list: rados -p cephfs_data -N restricted ls (empty output) Now I move my test file out of the restricted subdirectory and back in: mv /mnt/restricted/test2.bin /mnt/ sync mv /mnt/test2.bin /mnt/restricted/ sync No change: rados -p cephfs_data -N restricted ls (empty output) Next, I try moving the file out, and *copying* it back in: mv /mnt/restricted/test2.bin /mnt/ sync cp /mnt/test2.bin /mnt/restricted/ sync Now, I do see objects in the "restricted" namespace: rados -p cephfs_data -N restricted ls 10001fd.0001 10001fd. Also, if I create a *new* file in the "restricted" subdirectory, then its RADOS objects do end up in the correct namespace: dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M rados -p cephfs_data -N restricted ls 10001fe. 10001fe.0002 10001fe.0001 10001fd.0001 10001fd. By contrast, here's what happens to a non-empty *file,* when I try to set ceph.file.layout: dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin setfattr: test4.bin: Directory not empty So that fails, somewhat expectedly (although the error message is odd). In summary, there are two things that confuse me here: 1. Why does setting ceph.dir.layout.pool_namespace on a non-empty directory succeed, when setting ceph.file.layout.pool_namespace on a non-empty file fails (and even confusingly with a "Directory not empty" message)? 2. Considering that setting ceph.dir.layout.pool_namespace on a non-empty directory does succeed, why does mv'ing a file to a directory with a different pool_namespace, and then mv'ing it back, not result in its RADOS objects moving to the other namespace? So I'm curious: is this a bug (or two), or have I been misunderstanding what's actually the expected behaviour? Cheers, Florian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph orch is not working
thanks but it seems another issue. Is there any way to upgrade mgr without orchestrator ? or any other services ? We are online but cannot give any command to cluster alertmanager.ceph01 ceph01 *:9093,9094 running (10h) 6h ago 21M 63.1M- 0.25.0 c8568f914cd2 0bf1d544a775 ceph-exporter.ceph01 ceph01 running (10h) 6h ago 21M 59.0M- 17.2.6 c9a1062f7289 de3f438c448c ceph-exporter.ceph02 ceph02 running (10h) 6h ago 21M 59.5M- 17.2.6 c9a1062f7289 ba1edb035a66 crash.ceph01 ceph01 running (7h) 6h ago 21M 8640k- 17.2.6 2747c7f13104 e692b2314548 crash.ceph02 ceph02 running (7h) 6h ago 21M 8568k- 17.2.6 2747c7f13104 24535dbbfd76 crash.ceph03 ceph03 running (7h) 6h ago 21M 7394k- 17.2.6 2747c7f13104 86b981a9254c grafana.ceph01 ceph01 *:3000 running (10h) 6h ago 21M 212M- 9.4.7 954c08fa6188 7ef73e324096 mds.cephfs1.ceph01.niiovaceph01 stopped 6h ago 11h -- mds.cephfs1.ceph02.qsrlyoceph02 stopped 6h ago 21M -- mds.cephfs1.ceph03.kuysxqceph03 stopped 6h ago 21M -- mgr.ceph01.eydqvmceph01 *:8443,9283 running (7h) 6h ago 21M 439M- 17.2.6 2747c7f13104 6093b54fda30 mgr.ceph02.wwiqqsceph02 *:8443,9283 running (7h) 6h ago 21M 576M- 17.2.6 2747c7f13104 bd8a6a0e0dc7 mgr.ceph03.orwlyvceph03 *:8443,9283 running (7h) 6h ago 21M 438M- 17.2.6 2747c7f13104 61d563cc5fd2 mon.ceph01 ceph01 running (7h) 6h ago 21M 343M2048M 17.2.6 2747c7f13104 33c26b68b7e0 mon.ceph02 ceph02 running (7h) 6h ago 21M 324M2048M 17.2.6 2747c7f13104 479224b3c6d9 mon.ceph03 ceph03 running (7h) 6h ago 21M 314M2048M 17.2.6 2747c7f13104 affa57e31300 node-exporter.ceph01 ceph01 *:9100 running (10h) 6h ago 21M 37.5M- 1.5.0 0da6a335fe13 a868bdcacffb node-exporter.ceph02 ceph02 *:9100 running (10h) 6h ago 21M 36.5M- 1.5.0 0da6a335fe13 ffd26fa4c977 node-exporter.ceph03 ceph03 *:9100 running (10h) 6h ago 21M 38.3M- 1.5.0 0da6a335fe13 432fb5b9e903 osd.0ceph01 running (10h) 6h ago 21M 7501M9461M 17.2.8 259b35566514 3a451ff18bbf osd.1ceph02 running (10h) 6h ago 21M 8623M9589M 17.2.8 259b35566514 a420f4373061 osd.2ceph03 running (10h) 6h ago 7h 3270M9845M 17.2.8 259b35566514 57eba1a3fcaf osd.3ceph02 running (10h) 6h ago 7h 4580M9589M 17.2.8 259b35566514 6e4ffb81fee5 osd.4ceph03 running (10h) 6h ago 21M 3928M9845M 17.2.8 259b35566514 111b8d5ffb54 osd.5ceph01 running (10h) 6h ago 21M 11.1G9461M 17.2.8 259b35566514 50727e330c7a osd.6ceph02 running (10h) 6h ago 21M 8811M9589M 17.2.8 259b35566514 602b72c69ab6 osd.7ceph03 running (10h) 6h ago 7h 4491M9845M 17.2.8 259b35566514 1c16697e2c4c osd.8ceph01 error 6h ago 21M -9461M osd.9ceph02 running (10h) 6h ago 21M 9769M9589M 17.2.8 259b35566514 3e812c1dd841 osd.10 ceph03 running (10h) 6h ago 6h 2877M9845M 17.2.8 259b35566514 2102dcc50ead osd.11 ceph01 running (10h) 6h ago 21M 18.6G9461M 17.2.8 259b35566514 9422bb2e4dca osd.12 ceph02 running (10h) 6h ago 21M 24.8G9589M 17.2.8 259b35566514 736ae6924f2e osd.13 ceph03 running (6h) 6h ago 21M 40.0G9845M 17.2.8 259b35566514 b0fd0200dd0e osd.14 ceph01 running (10h) 6h ago 6h 4578M9461M 17.2.8 259b35566514 ec79e5a13a94 osd.15 ceph02 running (10h) 6h ago 21M 16.7G9589M 17.2.8 259b35566514 a9c876d93119 osd.16 ceph03
[ceph-users] Re: ceph orch is not working
I wonder if you are hitting the same issue as this. https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/D3PHDKKPL7LUHLAMSCO4Y5DESMKIA4FP/ If so, you might have to play around failing the manager so you can stop the upgrade. AFAIK downgrading versions is not something that is tested at all. On Wed, 5 Mar 2025 at 13:03, wrote: > Hi , we have issue using ceph orch , it doesn't send any command for real. > We try to restart or redeploy but nothing works > This happened downgrading from 17.2.8 to 17.2.6 > Thanks > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- Joshua Blanch Software Engineer Clyso GmbH p: +49 89 215252722 | a: Victoria, Canada w: https://clyso.com | e: joshua.bla...@clyso.com ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: upgrading ceph without orchestrator
I would even state updating ceph without the containers is easier. > > Is it possible to upgrade mgr without using ceph orch ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] ceph orch is not working
Hi , we have issue using ceph orch , it doesn't send any command for real. We try to restart or redeploy but nothing works This happened downgrading from 17.2.8 to 17.2.6 Thanks ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute
This is certainly intended behavior. If you checked the layout on the particular file, you would see it hasn’t changed. Directory layouts are the default for new files, not a control mechanism for existing files. It might be confusing, so we can talk about different presentations if there’s a better option (though this has been the behavior as long as we’ve had this functionality), but auto-migrating data is a significant challenge we haven’t sorted through and will have distinct controls if we ever manage to make it happen. -Greg On Wed, Mar 5, 2025 at 5:06 AM Florian Haas wrote: > Hello everyone, > > I'm seeing some behaviour in CephFS that strikes me as unexpected, and I > wonder if others have thoughts about it. > > Consider this scenario: > > * Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, > CephFS client is running kernel 5.15.0-133-generic. > * CephFS is mounted to /mnt, using a CephX identity that has rwp > permissions. > * CephFS is using a single data pool named cephfs_data. > * In the CephFS root directory, there is a subdirectory named > "restricted". That subdirectory currently has a single file, > "test2.bin", of 8 MiB. > > There is no other data in the data pool. I can verify this with the > following rados command: > > rados -n client.cephfs -p cephfs_data ls > 103.0001 > 103. > > Now I run this, from the root of my mounted CephFS: > > cd /mnt > setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted > > I realise this is naughty, because there is currently a file named > /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes > on a non-empty directory. However, the command does succeed, and I am > able to read back the ceph.dir.layout xattr: > > getfattr -n ceph.dir.layout restricted > # file: restricted > ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 > pool=cephfs_data pool_namespace=restricted" > > If at this stage I try to read RADOS objects in the "restricted" > namespace, I get an empty list: > > rados -p cephfs_data -N restricted ls > (empty output) > > Now I move my test file out of the restricted subdirectory and back in: > > mv /mnt/restricted/test2.bin /mnt/ > sync > mv /mnt/test2.bin /mnt/restricted/ > sync > > No change: > > rados -p cephfs_data -N restricted ls > (empty output) > > Next, I try moving the file out, and *copying* it back in: > > mv /mnt/restricted/test2.bin /mnt/ > sync > cp /mnt/test2.bin /mnt/restricted/ > sync > > Now, I do see objects in the "restricted" namespace: > > rados -p cephfs_data -N restricted ls > 10001fd.0001 > 10001fd. > > Also, if I create a *new* file in the "restricted" subdirectory, then > its RADOS objects do end up in the correct namespace: > > dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M > > rados -p cephfs_data -N restricted ls > 10001fe. > 10001fe.0002 > 10001fe.0001 > 10001fd.0001 > 10001fd. > > By contrast, here's what happens to a non-empty *file,* when I try to > set ceph.file.layout: > > dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M > > setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin > setfattr: test4.bin: Directory not empty > > So that fails, somewhat expectedly (although the error message is odd). > > In summary, there are two things that confuse me here: > > 1. Why does setting ceph.dir.layout.pool_namespace on a non-empty > directory succeed, when setting ceph.file.layout.pool_namespace on a > non-empty file fails (and even confusingly with a "Directory not empty" > message)? > > 2. Considering that setting ceph.dir.layout.pool_namespace on a > non-empty directory does succeed, why does mv'ing a file to a directory > with a different pool_namespace, and then mv'ing it back, not result in > its RADOS objects moving to the other namespace? > > So I'm curious: is this a bug (or two), or have I been misunderstanding > what's actually the expected behaviour? > > Cheers, > Florian > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] upgrading ceph without orchestrator
Is it possible to upgrade mgr without using ceph orch ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
