On Wed, 2008-12-10 at 12:10 -0800, Jim Searle wrote:
> Has there ever been a discussion about parallelizing rsync? We do
> some large file transfers across our wan which get done much faster if
> we run multiple rsync's at the same time. I wrote a perl script that
> traverses the directories and
Has there ever been a discussion about parallelizing rsync? We do some large
file transfers across our wan which get done much faster if we run multiple
rsync's at the same time. I wrote a perl script that traverses the directories
and figures out how many files to send with each rsync proces
Since it sounds like disk I/O is your limiting factor, you may wish to
look into updating multiple systems using a batch file. This requires
all the receiving systems to have identical files in the destination
hierarchy. You would first create a batch file by performing the
synchronization either
On Thu 01 Sep 2005, Xuehai Zhang wrote:
> >
> > If (similar) tasks are run in parallel, then the data of the files being
> > handled may still be in the buffer cache so that it doesn't need to get
> > read in from disk again. This will save time...
>
> I agree with you that caching effect might be
Paul,
Thanks for your response.
> > results. Why the time of transferring the file to 2N nodes is shorter than
> > twice of the time of transferring the same file to N nodes? Does it make
>
> If the network is not the bottleneck, then cpu or the disks are.
The network is 100Mbps LAN.
> If (simi
On Wed 31 Aug 2005, xuehai zhang wrote:
> results. Why the time of transferring the file to 2N nodes is shorter than
> twice of the time of transferring the same file to N nodes? Does it make
If the network is not the bottleneck, then cpu or the disks are. If
(similar) tasks are run in parallel
it make sense to you? What could be the reason if it makes sense in some way?
Thank you so much for your help in advance!
Xuehai
P.S. the script to do the parallel rsync
#!/bin/sh
LIST="ccn2"
if [ "$#" -gt "0" ] ; then
if [ "$1" -eq "2"
