As others have suggested, it's not in any sense a "secure erase" where data is overwritten. At the RADOS level, when an OSD receives a delete operation, it twiddles the metadata so the object isn't there any more, and BlueStore marks the space as unallocated, etc. You could go in and get it with disk forensics; I don't think we have any meaningful way of finding deleted objects with ceph-objectstore-tool or such (though I could be wrong?) but I don't think those are meaningfully different from a security perspective. There is absolutely no way for a RADOS client to retrieve deleted data using RADOS ops once a delete is performed (barring snapshots etc being in play).
RBD and CephFS have their own layers on top of this: rbd has a trash bin where objects sit for a while before being purged; files unlinked in CephFS get moved into a purge queue once there are no more active references to them, and that purge queue is processed by the mds to issue the actual object deletes. RGW has a similar purging concept but I don't know the details of it (it's quite sophisticated due to the semantics involved and the rules around S3 object versioning and how they're implemented.) As Janne suggested, use client-side encryption if you're worried about it. -Greg On Wed, May 28, 2025 at 6:43 AM Janne Johansson <icepic...@gmail.com> wrote: > > > > Pathological example: > > > > > > rbd rm $image (successful deletion) > > > ceph pause immediately after that > > > Do the recovery procedure noted above > > > > > > How likely is it that we would be able to recovery the data? > > > > Like most filesystems, pretty likely at a certain granularity. In the > > above case, the RBD trash system may even be in play. > > I’m confident that users can’t get at data, but to protect against thieves, > > liquidators etc., dmcrypt is your friend - just never get rid of the mon > > SSDs, or at the very least issue crypto erase operations on them before > > letting go. > > Yes, I agree with the above statement. If you have a client using this > RBD and needs to make sure evil ceph admins can't get to the data, > they should definitely encrypt the partition where this RBD gets > mounted. > Then it is just a matter of forgetting the key to this and delete the > RBD image as you please. > > If the secret you are hoping to protect is like a x509 cert key, it > will be around 1-2k at most, so anyone being able to read the raw > disks 5 seconds after you do "ceph pause" will then find this ------ > BEGIN KEY -------- and be able to steal this super important key > because it will be lying around somewhere on three or more OSDs. If > you as a client on the other hand LUKS-encrypt the RBD mount, then > wipe the key (or reset key to a random value you don't know > afterwards) and then delete the RBD image, the encrypted data will > possibly be able to be found by people with root access to OSD hosts - > or the physical boxes, but they will not be able to read the secret > data you wanted to protect. > > I'm somewhat of the idea that if someone asked me to successfully > piece together a 40G RBD image that was deleted some time ago on a > busy cluster, I would probably not be able to, not even to save my > kids lives. But others theoretically might, and auditors seldom care > for non-numeric values like "JJs kids lives" so if you have a real > case, just go with local disk or partition encryption and be done with > it. The perf hit is not so bad anymore and needing to trust someone > you might not trust and convince an auditor you can trust them is just > not worth the hassle. All major OSs have encrypted file system options > so this part is solved already, and if the data is important enough > for these kinds of questions you can take the burden of setting it up. > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
