Matt McCutchen wrote:
On Fri, 2007-12-28 at 00:15 -0500, Eric S. Johansson wrote:
it is possible, I've seen it done, but I can't find the library/tool anymore.
I'm curious: what was the nature of this tool (if you remember)? A
modified version of rsync? A dispersed storage service with an rsync
daemon interface? A virtual filesystem? Did delta transfers work
properly?
Unfortunately, my memory is somewhat foggy on the finer points of this toolkit.
What I remember is that it split up the backup image into N parts and you only
needed to recover M in order to reconstruct your data set. I seem to remember
it used something analogous to rsync to minimize unnecessary duplication but,
that's only a faint memory. I believe it was orders of magnitude better than
what we have today and it's a real shame it never caught on.
again with a pre and post processing capability, we could add that functionality
in without modding the baseline
It's not clear to me what kind of pre- or postprocessing capability you
are thinking of that would make dispersed storage with rsync practical.
A per-file approach with --source-filter and --dest-filter would have
the same disadvantages as per-file encryption, and by the time you
arrange for the retrieval filter to combine multiple files, I don't see
what you gain by having rsync in the picture. Using rsync with a
virtual filesystem that implements the dispersed storage is more
natural.
What seemed like a bright idea during composition appears more like a rusty
clunker after one hits the send key. :-) The original, albeit poorly thought
out, idea was to replicate the directory structure on all destination machines,
then take each file, split into M parts and then replicate each part into the
right portion of the filesystem hierarchy on remote machines. Yes, not the best
idea I've ever had. I think the virtual filesystem does make a lot more sense.
How would you implement caching so that you only need to scan the local
filesystem once instead of every time you compare it against the remote file system?
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
