Silly suggestion, perhaps, but...
Ever considered simply breaking your sync down into several separate sets? IE instead of rsync /foo/baz [EMAIL PROTECTED]:/foo, where /foo/baz contains ten directories each then spiraling down into hundreds of thousands more... rsync /foo/baz/1 [EMAIL PROTECTED]:/foo/baz, then rsync /foo/baz/2 [EMAIL PROTECTED]:/foo/baz, and so on. Enterprising sorts can, of course, write wrappers in Perl or the shell of your choice to automatically sync everything in /foo/baz one directory and/or file at a time with ease.
I haven't personally run into your upper end limits on heavy-duty equipment, but I've used the technique described above to keep old 32MB and 64MB machines in successful service as rsync servers; the principle is the same - scale your single sync down to a level that your hardware can handle.
(It should also be possible to make rsync more memory-friendly when the size of the job exceeds the resources available, by causing it to do the exact same thing as the wrapper described above does, but I'm not going to kvetch.)
Jim Salter JRS Systems
Hi, folks.
We've gone where no man has gone before. On HP-UX, rsync bombs at about 1.75 million directories, files and links (combined sum) in a single transfer.
Is there a professional-grade alternative on HP-UX for folks willing to pay for it? It wouldn't even need to be network-aware, just from single-system areas to the same box, but with the nifty delete and update features that rsync has. My searches turn up unison and some other tools (BSD mirror, etc.), but rsync has beaten any other open-source solution hands down on the scalability side of things. Now, we need more ...
Thanks,
A. Daniel King, System Analyst HP-UX, Linux, Solaris
-- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
