Hi, I agree with Daniel, but I have been successfully using this type of backup strategy for two years between two CEPH clusters, with 1.6 PB and over 1 billion files/folders.
A few comments: 1) Using the rctime attribute is better than nothing. 2) The rctime=01/01/2300 issue is not really a problem. I run non-recursive rsyncs on existing folders that have an rctime > previous backup date. In the case of rctime=01/01/2300, this results in unnecessary rsyncs, but in reality, they are very fast. So far, I have not observed any degradation in backup times. 3) I take a snapshot before each backup, and the date of the previous snapshot gives me the rctime reference value of the folders/files to be backed up (so no need to keep the rctime values in a file like Eric does). 4) fpart/fpsync is my best friend for transferring, with parallel rsyncs, new folders that have more than 1000 entries (ceph.dir.rentries attribute). Every evening, I generate a hundred backup jobs (1 job for each project directory with 20 CPUs allocated to each job for parallel rsyncs) that are executed on an HPC cluster in a specific queue with only 3 nodes. There is quite a lot of user activity during the day and the average backup time is around 2 hours. Obviously, this depends on the volume to be transferred and/or deleted on the target cluster. For now, everything is working well, but the main limitation of this strategy remains the management of directory moves/renames (creation of the new directory and deletion of the old one). Kind regards, Arnaud Le 13/06/2025 08:58, « Burkhard Linke » <burkhard.li...@computational.bio.uni-giessen.de <mailto:burkhard.li...@computational.bio.uni-giessen.de>> a écrit : Hi, On 12.06.25 21:58, Daniel Vogelbacher wrote: > Hi Eric, > > On 6/12/25 17:33, Eric Le Lay wrote: >> I use rsync to copy data (~10TB) to backup storage. >> >> To speed things up I use the ceph.dir.rctime extended attribute to >> instantly ignore sub-trees that haven't changed without iterating >> through their contents. >> I have to maintain the ceph.dir.rctime value between backups: I just >> keep it in a file per top-level directory on the target storage. > This sounds interesting. Can you give some advice how to set this up > with rsync? the rctime attribute is useful, but I wouldn't rely on it. As far as I know it is stored in a directory inode, so each operation on a file or directory will update the rctime on all path elements (not sure whether this happens synchronously or asynchronously). The problem is the fact that it is just a single value. Imagine one rogue user or rogue host that touches a file in a subdirectory, set the ctime to 01/01/2300, and then removes the file. Although the removal is the last operation, setting the ctime will also update the rctime of all path elements. And the removal if the file cannot revert this. So your backup check will detect a last change in 01/01/2300 for the subtree and probably performs a complete rsync. Even if all files are still the same. rctime is fine for controlled environments without rogue elements (== no users... ;-) ). And it can definitely be used to skip subtrees. But the check can easily be rendered useless. Best regards, Burkhard > > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io <mailto:ceph-users-le...@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
