> I would like to get a better understanding at what happens under the hood > when an object is deleted in a Ceph cluster. > > Is there a (detailed) write up of the deletion process?
I suspect that such would quickly become obsolete once written but would be good to know. > > I would like to be able to answer questions like: > > - How can we be sure that after the successful completion of for example a > "rbd rm $image" command it is impossible for the data to be restored. RBD images themselves are thin-provisioned usually, which takes care of that angle. > > - How do we know we don't get deleted data returned that was marked as > deleted (but never overwritten with new data) > > Can we be sure? Or are there ways to perform a recovery procedure (as a > storage admin with access to all components of the cluster), i.e. gather > information about higher level Ceph abstractions (rbd, file in CephFS, RGW > bucket) that have existed in the past, gather information about what objects > where in use, do CRUSH calculation (and consult upmaps) to obtain OSDs > involved, Query RocksDB / Bluestore to obtain information about LBA on disk, > read the data and in this way reconstruct the object(s). Or at a smaller granularity, dd the OSD devices. I once had to effectively do that in college to get an art student’s paper back after a certain crappy piece of software MINCEd her floppy. > > Pathological example: > > rbd rm $image (successful deletion) > ceph pause immediately after that > Do the recovery procedure noted above > > How likely is it that we would be able to recovery the data? Like most filesystems, pretty likely at a certain granularity. In the above case, the RBD trash system may even be in play. I’m confident that users can’t get at data, but to protect against thieves, liquidators etc., dmcrypt is your friend - just never get rid of the mon SSDs, or at the very least issue crypto erase operations on them before letting go. Were Ceph to zero data on delete, that would present substantially more IOPs to the devices, and in the case of SSDs erode endurance. Certain industries might pay the price nonetheless. I must note as an aside that dding /dev/zero to an SSD most likely does not overwrite data. > > I understand it's hard to put a number on it, so pointers to where (parts) of > this process is described will help me getting a better understanding. I'm > also aware that it highly depends on how busy a cluster would be, what type > of media involved, if the discard option of Ceph and disks is in use, and > many other factors. > > Thanks, > > Stefan > > > _______________________________________________ > Dev mailing list -- d...@ceph.io > To unsubscribe send an email to dev-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
