Re: How to speed up rsync when haveing lots of files

On Wed, Mar 4, 2009 at 1:26 AM, Boniforti Flavio wrote: > Is there any way to know *in advance* if using or not using "-z" could > be the better solution? I don't think so. You need to run your tasks on your own hardware and network to see where the bottlenecks are. If the task is CPU-bound, tur

Re: How to speed up rsync when haveing lots of files

On Wed 04 Mar 2009, Boniforti Flavio wrote: > > > This question has been answered. Upgrade to 3.0.5 on both sides. > > Why has upgrading to 3.0.5 to be done *on both sides*? Any explanation?! Because the improved algorithm will only work if both sides understand it... > As I'm running Debian Le

RE: How to speed up rsync when haveing lots of files

On Wed, 2009-03-04 at 13:41 +0100, Boniforti Flavio wrote: > > This question has been answered. Upgrade to 3.0.5 on both sides. > > Why has upgrading to 3.0.5 to be done *on both sides*? Any explanation?! > As I'm running Debian Lenny (which has 3.0.3), do I have to switch to > Debian Sid to get i

Re: How to speed up rsync when haveing lots of files

On 3/4/2009, Boniforti Flavio (fla...@piramide.ch) wrote: > Why has upgrading to 3.0.5 to be done *on both sides*? Any explanation?! It is quite simple... support for the new options requires 3.0.x on both sides... > As I'm running Debian Lenny (which has 3.0.3), do I have to switch to > Debian S

RE: How to speed up rsync when haveing lots of files

> This question has been answered. Upgrade to 3.0.5 on both sides. Why has upgrading to 3.0.5 to be done *on both sides*? Any explanation?! As I'm running Debian Lenny (which has 3.0.3), do I have to switch to Debian Sid to get it upgraded? Thanks, Flavio Boniforti PIRAMIDE INFORMATICA SAGL Via

Re: How to speed up rsync when haveing lots of files

On 3/4/2009, Daniel.Li (daniel...@usish.com) wrote: > So I wanna know how to maximize the performance both of large file > (1GB patter file) and small file (MP3, about 2~5MB per file) > > Maybe MP3 shouldn't be considered small files, right? > > Do u guys have any suggestions? > Any hint or adv

Re: How to speed up rsync when haveing lots of files

On Tue, 2009-03-03 at 23:46 -0700, lewis butler wrote: > On 3-Mar-2009, at 22:55, Daniel.Li wrote: > > -z is apparently affect the performance when CPU has a lower > > frequency, > > like 200MHz or so. When doing rsync, 100% cpu occupied, which limits > > network performance. > > You should not

RE: How to speed up rsync when haveing lots of files

> You should not use -z at ALL with large numbers of small > files. The increased latency of the > compression/decompression will far exceed any time saved in > transmission. You're not on a 300 baud modem, I assume? > -z should be used when you are sending large files that are > compressib

Re: How to speed up rsync when haveing lots of files

On 3-Mar-2009, at 22:55, Daniel.Li wrote: -z is apparently affect the performance when CPU has a lower frequency, like 200MHz or so. When doing rsync, 100% cpu occupied, which limits network performance. You should not use -z at ALL with large numbers of small files. The increased latency

Re: How to speed up rsync when haveing lots of files

On Tue, 2009-03-03 at 21:26 -0700, lewis butler wrote: > On 3-Mar-2009, at 20:33, Daniel.Li wrote: > > On Tue, 2009-03-03 at 08:58 -0500, Charles Marcus wrote: > >> On 3/3/2009 8:52 AM, m...@bortal.de wrote: > > unfortunatelly rsync is beeing REALLY slow and produces a high > > load when

Re: How to speed up rsync when haveing lots of files

On 3-Mar-2009, at 20:33, Daniel.Li wrote: On Tue, 2009-03-03 at 08:58 -0500, Charles Marcus wrote: On 3/3/2009 8:52 AM, m...@bortal.de wrote: unfortunatelly rsync is beeing REALLY slow and produces a high load when we try to sync lots of files (>250 000 small files). What version of rsync

Re: How to speed up rsync when haveing lots of files

On Tue, Mar 3, 2009 at 9:33 PM, Daniel.Li wrote: >> Upgrade to 3.0.5 (on both ends) > > OK, besides this, is there any other way to improve the network > performance? some thing like change the option or what? This is a well-known performance issue in rsync that was specifically addressed in v3.0

Re: How to speed up rsync when haveing lots of files

On Tue, 2009-03-03 at 08:58 -0500, Charles Marcus wrote: > On 3/3/2009 8:52 AM, m...@bortal.de wrote: > >>> unfortunatelly rsync is beeing REALLY slow and produces a high load when > >>> we try to sync lots of files (>250 000 small files). > > >> What version of rsync are you using? > > > # rsync

Re: How to speed up rsync when haveing lots of files

On 3/3/2009 8:52 AM, m...@bortal.de wrote: >>> unfortunatelly rsync is beeing REALLY slow and produces a high load when >>> we try to sync lots of files (>250 000 small files). >> What version of rsync are you using? > # rsync --version > rsync version 2.6.9 protocol version 29 Upgrade to 3.0.

Re: How to speed up rsync when haveing lots of files

# rsync --version rsync version 2.6.9 protocol version 29 Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others. Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, ACLs, 64-bit system inums

Re: How to speed up rsync when haveing lots of files

m...@bortal.de wrote: > unfortunatelly rsync is beeing REALLY slow and produces a high load when > we try to sync lots of files (>250 000 small files). What version of rsync are you using? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change option