Hi Frédéric, > Can you try the below command? > > $ rados -p mailbox listsnaps rbd_data.26f7c5d05af621.0000000000002adf > rbd_data.26f7c5d05af621.0000000000002adf: > cloneid snaps size overlap > 3 3 4194304 [] > head - 4194304 <---- Do you see this line?
$ rados -p mailbox listsnaps rbd_data.26f7c5d05af621.0000000000002adf rbd_data.26f7c5d05af621.0000000000002adf: cloneid snaps size overlap 27534 27523,27534 4194304 [0~1904640] 27553 27553 4194304 [0~1904640] 27636 27599,27636 4194304 [368640~3825664] 27673 27673 4194304 [1126400~3067904] 27710 27710 4194304 [1646592~2547712] 27721 27721 4194304 [1634304~2560000] 27732 27732 4194304 [1695744~2498560] 27743 27743 4194304 [1695744~2498560] 27780 27780 4194304 [] And $ rados -p cephfs_data_ec listsnaps 100198218f2.0000008a 100198218f2.0000008a: cloneid snaps size overlap 4762 4705 4194304 [] > If you don't see the 'head' line, then you're probably facing the orphan > clones AKA leaked snapshots bug described here [1], that was fixed by [2]. Looks like we are affected by https://tracker.ceph.com/issues/64646 on both pools. We are currently on 17.2.7, while the fix is in 17.2.8. > To get rid of these orphan clones, you need run to run the below command on > the pool that we requeue orphan objects for being snap trimmed. See [3] for > details. > > $ ceph osd pool force-remove-snap cephfs_data_ec > > Note that : > > 1. there's a --dry-run option you can use. > 2. This command should only be run ONCE. Running it twice or more is useless > and can lead to OSDs crashing (restarting fine, but still... crashing at the > same time). This sounds a little bit scary to be honest. Do you mean osds can crash once and restart fine or do they crash, restart, crash, … ? > > Let us know how it goes. > > Cheers, > Frédéric. > > [1] https://tracker.ceph.com/issues/64646 > [2] https://github.com/ceph/ceph/pull/55841 > [3] https://github.com/ceph/ceph/pull/53545 > > ----- Le 31 Jan 25, à 9:54, Frédéric Nass frederic.n...@univ-lorraine.fr a > écrit : > >> Hi Felix, >> >> This is weird. The most likely explanation is that this resulted from a bug >> that >> has certainly been fixed. >> >> What you try could as a starter is a ceph-bluestore-tool fsck [1] over the >> primary OSD of rbd_data.26f7c5d05af621.0000000000002adf to see if it lists >> any >> inconsistencies: >> >> $ cephadm shell --name osd.$i --fsid $(ceph fsid) ceph-bluestore-tool fsck >> --path /var/lib/ceph/osd/ceph-$i/ --deep yes >> >> If it doesn't then I suppose you'll have to get rid of these objects you >> can't >> 'rados rm' with the help of ceph-objectstore-tool <remove> [2]. >> >> Regards, >> Frédéric. >> >> [1] https://docs.ceph.com/en/latest/man/8/ceph-bluestore-tool/ >> [2] >> https://docs.ceph.com/en/latest/man/8/ceph-objectstore-tool/#removing-an-object >> >> ----- Le 31 Jan 25, à 8:29, Felix Stolte <f.sto...@fz-juelich.de> a écrit : >> >>> Hi Frederic, >>> thanks for you suggestions. I took a look at all the objects and discovered >>> the >>> following: >> >>> 1. rbd_id.<image_name>, rbd_header.<image_id> and rbd_object_map.<image_id> >>> exist only for the 3 images listed by ‚rbd ls‘ (i created a test image >>> yesterday). >> >>> 2. There are about 30 different image_ids with rbd_data.<image_id> Objects >>> which >>> do not have an id, header oder object_map object >> >>> After that i tried to get the stats of one of the orphaned objects: >> >>> rados -p mailbox stat rbd_data.26f7c5d05af621.0000000000002adf >>> error stat-ing mailbox/rbd_data.26f7c5d05af621.0000000000002adf: (2) No such >>> file or directory >>> I double checked, that object name is the one listed by 'rados ls‘. What >>> makes >>> it worse is that i neither can stat, get or rm the objects while they still >>> are >>> counted for disk usage. We will remove the whole pool for sure, but i really >>> like to get to the cause of this to prevent it from happening again. >> >>>> Am 30.01.2025 um 10:54 schrieb Frédéric Nass >>>> <frederic.n...@univ-lorraine.fr>: >> >>>> Hi Felix, >> >>>> Every rbd_data object belongs to an image that should have : >> >>>> - an rbd_id.<image_name> object containing the image id (that you can get >>>> with >>>> rbd info) >>>> - an rbd_header object with omap atrributes you can list with listomapvals >> >>>> To identify the image name these rbd_data object belong(ed) to you could >>>> list >>>> all rbd_id objects in that pool and, for each one of them, print the image >>>> id >>>> and the image name with the below command: >> >>>> $ for rbd_id in $(rados -p $poolname ls | grep rbd_id) ; do echo "$(echo >>>> $rbd_id >>>> | cut -d '.' -f2) : $(rados -p $poolname get $rbd_id - | strings)" ; done >>>> image2 : 2a733b30debc84 >>>> image1 : 28d4fc1dddd922 >> >>>> It might take some time but you'd get a clearer view of what these rbd_data >>>> objects refer(ed) to. Also, if you can decode the timestamps in 'rados -p >>>> rbd >>>> listomapvals rbd_header.<id> | strings' output, you could know when each >>>> image >>>> was created and accessed for the last time. >> >>>> Hope that helps. >> >>>> Regards, >>>> Frédéric. >> >>>> PS: If you're moving away from iSCSI and only have 2 remaining images in >>>> this >>>> pool, you may also wait until these images are no longer in use and then >>>> detach >>>> them and remove the whole pool. >> >>>> ----- Le 30 Jan 25, à 9:09, Felix Stolte <f.sto...@fz-juelich.de> a écrit : >> >>>>> Hi Frederic, >>>>> there is no namespace. The pool in question has the application rbd, but >>>>> is not >>>>> the default pool named ‚rbd' >> >>>>>> Am 29.01.2025 um 11:24 schrieb Frédéric Nass >>>>>> <frederic.n...@univ-lorraine.fr>: >> >>>>>> Hi Felix, >> >>>>>> Any RADOS namespaces in that pool? You can check using either: >> >>>>>> rbd namespace ls rbd >> >>>>>> or >> >>>>>> rados stat -p rbd rbd_namespace && rados -p rbd listomapvals >>>>>> rbd_namespace >> >>>>>> The rbd_data objects might be linked to namespaced images that can only >>>>>> be >>>>>> listed using the command: rbd ls --namespace <namespace> >>>>>> I suggest checking this because the 'rbd' pool has historically been >>>>>> Ceph's >>>>>> default RBD pool, long before iSCSI began using it (in its hardcoded >>>>>> implementation). >> >>>>>> Might be worth checking this before taking any actions. >> >>>>>> Regards, >>>>>> Frédéric. >> >>>>>> ----- Le 29 Jan 25, à 8:53, Felix Stolte f.sto...@fz-juelich.de a écrit : >> >>>>>>> Hi Alexander, >> >>>>>>> trash is empty and rbd ls only lists two images with the prefix >>>>>>> rbd_data.1af561611d24cf and rbd_data.ed93e6548ca56b >> >>>>>>> rados ls gives: >> >>>>>>> rbd_data.d1b81247165450.00000000000055d2 >>>>>>> rbd_data.32de606b8b4567.0000000000012f2f >>>>>>> rbd_data.ed93e6548ca56b.00000000000eef03 >>>>>>> rbd_data.26f7c5d05af621.0000000000002adf >>>>>>> …. >> >>>>>>> Am 28.01.2025 um 22:46 schrieb Alexander Patrakov <patra...@gmail.com>: >> >>>>>>> Hi Felix, >> >>>>>>> A dumb answer first: if you know the image names, have you tried "rbd >>>>>>> rm $pool/$imagename"? Or, is there any reason like concerns about >>>>>>> iSCSI control data integrity that prevents you from trying that? >> >>>>>>> Also, have you checked the rbd trash? >> >>>>>>> On Tue, Jan 28, 2025 at 5:43 PM Stolte, Felix <f.sto...@fz-juelich.de> >>>>>>> wrote: >> >>>>>>> Hi guys, >> >>>>>>> we have a rbd pool we used for images exported via ceph-iscsi on a >>>>>>> 17.2.7 >>>>>>> cluster. The pool uses 10 times the diskspace i would suppose it should >>>>>>> and >>>>>>> after investigating we noticed a lot of rbd_data Objects which images >>>>>>> are no >>>>>>> longer present. I assume that the original images were deleted using >>>>>>> the gwcli >>>>>>> but not all Objects have been removed properly. >> >>>>>>> What would be the best/most secure way to get rid of these orphaned >>>>>>> objects and >>>>>>> reclaim the diskspace? >> >>>>>>> Best regards >>>>>>> Felix >> >>>>>>> --------------------------------------------------------------------------------------------- >>>>>>> --------------------------------------------------------------------------------------------- >>>>>>> Forschungszentrum Juelich GmbH >>>>>>> 52425 Juelich >>>>>>> Sitz der Gesellschaft: Juelich >>>>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>>>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>>>>> --------------------------------------------------------------------------------------------- >>>>>>> --------------------------------------------------------------------------------------------- >> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>>>>> -- >>>>>>> Alexander Patrakov >> >>>>>>> mit freundlichem Gruß >>>>>>> Felix Stolte >> >>>>>>> IT-Services >> >>>>>>> --------------------------------------------------------------------------------------------- >>>>>>> --------------------------------------------------------------------------------------------- >>>>>>> Forschungszentrum Juelich GmbH >>>>>>> 52425 Juelich >>>>>>> Sitz der Gesellschaft: Juelich >>>>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>>>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>>>>> --------------------------------------------------------------------------------------------- >>>>>>> --------------------------------------------------------------------------------------------- >> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>>> mit freundlichem Gruß >>>>> Felix Stolte >> >>>>> IT-Services >> >>>>> --------------------------------------------------------------------------------------------- >>>>> --------------------------------------------------------------------------------------------- >>>>> Forschungszentrum Juelich GmbH >>>>> 52425 Juelich >>>>> Sitz der Gesellschaft: Juelich >>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>>> --------------------------------------------------------------------------------------------- >>>>> --------------------------------------------------------------------------------------------- >> >>> mit freundlichem Gruß >>> Felix Stolte >> >>> IT-Services >> >>> --------------------------------------------------------------------------------------------- >>> --------------------------------------------------------------------------------------------- >>> Forschungszentrum Juelich GmbH >>> 52425 Juelich >>> Sitz der Gesellschaft: Juelich >>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>> --------------------------------------------------------------------------------------------- >>> --------------------------------------------------------------------------------------------- >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io mit freundlichem Gruß Felix Stolte IT-Services --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
