Hi, On Tue, 10 Aug 2004, James Bagley Jr wrote:
> I've had some problems using rsync to transfer directories with more than > 3 million files. Here's the error message from rsync: > > <snip> > ERROR: out of memory in receive_file_entry > rsync error: error allocating core memory buffers (code 22) at util.c(116) > rsync: connection unexpectedly closed (63453387 bytes read so far) > rsync error: error in rsync protocol data stream (code 12) at io.c(342) > </snip> > > I'm doing a pull on a linux system fom the HP-UX system that actually > houses the data. Both using rsync-2.6.2. The one solution i've come up > with isn't pretty, but seems to work. How is your RAM/swap situation on the linux side? I am rsyncing the 4 million files of ftp.gwdg.de to a backup server each night, one-shot, no problems. It just needs 7 or more hours... rsync[28526] (receiver) heap statistics: arena: 233472 (bytes from sbrk) ordblks: 9 (chunks not in use) smblks: 3 hblks: 1282 (chunks from mmap) hblkhd: 357650432 (bytes from mmap) allmem: 357883904 (bytes from sbrk + mmap) usmblks: 0 fsmblks: 96 uordblks: 71072 (bytes used) fordblks: 162400 (bytes free) keepcost: 135048 (bytes in releasable chunk) Number of files: 4099094 Number of files transferred: 27160 Total file size: 1568423288614 bytes Total transferred file size: 16916178416 bytes Literal data: 10758972422 bytes Matched data: 6158385953 bytes File list size: 119674315 Total bytes written: 11990441 Total bytes read: 10885842335 wrote 11990441 bytes read 10885842335 bytes 410271.35 bytes/sec total size is 1568423288614 speedup is 143.92 rsync[26637] (generator) heap statistics: arena: 233472 (bytes from sbrk) ordblks: 8 (chunks not in use) smblks: 3 hblks: 1285 (chunks from mmap) hblkhd: 357924864 (bytes from mmap) allmem: 358158336 (bytes from sbrk + mmap) usmblks: 0 fsmblks: 96 uordblks: 90760 (bytes used) fordblks: 142712 (bytes free) keepcost: 135048 (bytes in releasable chunk) ==== end ======== RC=0 ============ 040810.0100 040810.0822 Cheers -e -- Eberhard Moenkeberg ([EMAIL PROTECTED], [EMAIL PROTECTED]) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
