Hello,
I've had some problems using rsync to transfer directories with more than
3 million files. Here's the error message from rsync:
<snip>
ERROR: out of memory in receive_file_entry
rsync error: error allocating core memory buffers (code 22) at util.c(116)
rsync: connection unexpectedly closed (63453387 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(342)
</snip>
I'm doing a pull on a linux system fom the HP-UX system that actually
houses the data. Both using rsync-2.6.2. The one solution i've come up
with isn't pretty, but seems to work.
Basically, I wrote a shell function that runs an rsync process for each
subdirectory if necessary. I'm using "find | wc -l" to count the number
of files in the source path and then calling the function again for each
subdirectory if the number is more than 2mil. Perhaps recursion is a bad
idea? It's the only way I could think of to catch the case where all the
files exist in a single directory several levels below the top. Anyways,
i'll take the function out of my script and paste it here but it may not
work right taken out of context. YMMV.
<snip>
function DC {
# "Divide and Conquer"
# evil hack to get around rsync's 3mil file limit
# Accepts two arguments as the srcpath and dstpath. Will not work
# if srcpath is local.
srchost=`echo $1 | awk -F: '{ print $1 }'`
srcdir=`echo $1 | awk -F: '{ print $2 }'`
num_files=`ssh [EMAIL PROTECTED] "find $srcdir | wc -l"`
if [ $((num_files)) -gt 2000000 ]
then
echo "WARNING! file count greater than 2mil, recursing into subdirs."
for file in `ssh [EMAIL PROTECTED] "ls $srcdir"`
do
dstpath=`echo $2/$file`
DC $1/$file $dstpath
done
else
rsync $rsync_opts $1 $2
fi
}
</snip>
Comments? Better ideas?
--
James Bagley | CDI Innovantage
[EMAIL PROTECTED] | Technical Computing UNIX Admin Support
DON'T PANIC | Agilent Technologies IT
--
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
