On Mon, Mar 24, 2025 at 3:17 PM Venky Shankar <vshan...@redhat.com> wrote: > > [cc Jos] > > Hi Alexander, > > I have a few questions/concerns about the list mentioned below: > > On Fri, Mar 21, 2025 at 11:09 AM Alexander Patrakov <patra...@gmail.com> > wrote: > > > > Hello Vladimir, > > > > Please contact croit via https://www.croit.io/contact for unofficial > > (not yet fully reviewed) patches and mention me. They are currently > > working on a similar problem and have solved at least the following: > > > > * Crashes when two directories are being mirrored in parallel > > Are you talking about bi-directional mirroring for the same directory > b/w two ceph clusters?
No, I am talking about https://github.com/ceph/ceph/pull/60667, already merged. Double-close, thus unexpectedly closing the fd open by another thread => crash. > > > * Data corruption (snapshots on the destination having missing files > > or files "from the future") due to incorrect cache maintenance in > > libcephfs > > There have been bugs identified in the mirror daemon and the snapdiff > library. These have been fixed and merged in the main branch (or under > review) waiting to get backported to releases. True. > > > * Confusion that results in the failed mirroring when a directory is > > replaced by a symlink > > This is a bug in the mirror daemon when the sync mechanism uses > snapdiff -- fix is under review. > > > * Useless work done to mirror old snapshots that nobody needs, while > > the latest snapshot has the highest business value > > OK. So this is more of an enhancement (that you proposed) than a bug > to avoid the mirror daemon starting synchronizing snapshots from the > very first and also possibly synchronize the latest snapshot first > since that snapshot has the most business value. True. > > > * Useless stat() calls slowing down the mirroring significantly > > There are some (not many) unnecessary stats in the scan code path, > however, most of those stat requests should be satisfied from the > client's cache. If you have details on the slowness, please create a > tracker with the details. Those unnecessary stats should be done away > with irrespective of the requests satisfied from clients cache... Jos? That's interesting, as we had to tune the cache later for a different reason (bug identical to https://tracker.ceph.com/issues/58640#note-11 but affecting patched cephfs-mirror, not ceph-fuse) to avoid reads saturating the interface without any corresponding writes. > > > * They also introduced multi-threaded mirroring so that more than one > > file is copied at a time and more than one subdirectory is scanned at > > a time - this helps with directories containing many small files > > Again, this is an enhancement feature being developed by Croit. A > change has been posted for review. > > > > > Due to the above, I absolutely cannot recommend using the unpatched > > cephfs-mirror and would rather (if you don't want to contact croit for > > patches) ask you to switch to rsync with some wrapper script that > > runs, in parallel, multiple copies operating on different > > subdirectories. > > The *real* slowness in the mirror daemon is the data sync for regular > files over a slow link b/w clusters. This is due to the fact that the > mirror daemon transfers the whole file even if only some bytes have > been modified. As you can see, this is inefficient and scales badly > with large files. For tentacle release, the MDS is introducing > blockdiff operation support -- to be able to get a list of changes > blocks for a file between snapshots and the mirror daemon would make > use of this. This enhancement will also be backported along with other > enhancements by Croit, so latest ceph releases should see a vast > improvement in synchronization time. Well, let me disagree. While the scenario above is definitely relevant, what we hit is completely different and would also manifest on a 100-gigabit network with zero latency and with the proposed blockdiff optimization applied. The HDD-based cluster has, in some folders, many small files written by many CephFS clients. As cephfs-mirror in its original version has only one sequential thread per directory, it is limited by Q1 IOPS of the cluster: it does not start reading a file until it finishes with the previous one. Therefore, it is limited by the seek latency of HDDs. For the same reason, it is limited by the throughput of one HDD per configured-to-mirror directory. What the patch does is transfer multiple files from the same directory in parallel (thus keeping more than HDD at a time busy), and, in later revisions, also scanning multiple subdirectories of a configured-to-mirror directory in parallel. > > > > > On Fri, Mar 21, 2025 at 12:48 AM Vladimir Cvetkovic > > <vladimir.cvetko...@hostpapa.com> wrote: > > > > > > Hi everyone, > > > > > > We have two remote clusters and are trying to set up snapshot mirroring. > > > Our directories are in a syncing state but never seem to actually sync > > > the > > > snapshots. > > > Snapshot mirroring seems way to slow for large file systems - does anyone > > > have a working setup? > > > > > > Thanks! > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > > > -- > > Alexander Patrakov > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > -- > Cheers, > Venky > -- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
