Re: Parallel rsync's for better Performance.

2009-10-28 Thread Matt McCutchen
On Wed, 2009-10-28 at 23:46 +0100, Matthias Schniedermeyer wrote: > On 28.10.2009 18:27, Matt McCutchen wrote: > > On Wed, 2009-10-28 at 17:24 +0100, Matthias Schniedermeyer wrote: > > > On 28.10.2009 10:35, Matt McCutchen wrote: > > > > On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wr

Re: Parallel rsync's for better Performance.

2009-10-28 Thread Matthias Schniedermeyer
On 28.10.2009 18:27, Matt McCutchen wrote: > On Wed, 2009-10-28 at 17:24 +0100, Matthias Schniedermeyer wrote: > > On 28.10.2009 10:35, Matt McCutchen wrote: > > > On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wrote: > > > > Otherwise parallel rsyncs completly kill any performance you

Re: Operation not supported (95)

2009-10-28 Thread Matt McCutchen
On Wed, 2009-10-28 at 18:55 +, Kay wrote: > I'm running into a problem that I'm sure which part of the process > fails. I use wget (OSX Leopard, wget 3.0.6 compiled from source) to > backup my ~/Pictures directory to a linux server, also with wget 3.0.6 > (Debian). In persuit of being sur

Re: Parallel rsync's for better Performance.

2009-10-28 Thread Hendrik Visage
On Wed, Oct 28, 2009 at 6:24 PM, Matthias Schniedermeyer wrote: > No. That's a fundamental problem with ANY rotating media device. > > I don't say say that you can't build something for the people that have > that kind of hardware, or that are constrainted by high bandwidth & > latency network con

Re: Parallel rsync's for better Performance.

2009-10-28 Thread Matt McCutchen
On Wed, 2009-10-28 at 17:24 +0100, Matthias Schniedermeyer wrote: > On 28.10.2009 10:35, Matt McCutchen wrote: > > On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wrote: > > > Otherwise parallel rsyncs completly kill any performance you had because > > > normal HDDs will fall into a see

Operation not supported (95)

2009-10-28 Thread Kay
Hi guys, I'm running into a problem that I'm sure which part of the process fails. I use wget (OSX Leopard, wget 3.0.6 compiled from source) to backup my ~/Pictures directory to a linux server, also with wget 3.0.6 (Debian). In persuit of being sure I get everything from HFS filesystem (re

Re: OS X 10.6 (Snow Leopard) HFS+ File Compression

2009-10-28 Thread Mike Bombich
HFS compression can be preserved as long as the relevant xattr(s) and flags on those files are preserved.  A compressed file has the compressed data in a hidden xattr (com.apple.decmpfs if < 4Kb, com.apple.ResourceFork if more), and has the UF_COMPRESSED flag set (decimal 40).  When rsync encounter

Re: Parallel rsync's for better Performance.

2009-10-28 Thread Matthias Schniedermeyer
On 28.10.2009 10:35, Matt McCutchen wrote: > On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wrote: > > On 28.10.2009 09:05, Satish Shukla wrote: > > > We have huge data to sync usually everyday and I wish rsync could > > > guarantee performance. > > > > > > I thought of spliting the d

Re: Parallel rsync's for better Performance.

2009-10-28 Thread Matt McCutchen
On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wrote: > On 28.10.2009 09:05, Satish Shukla wrote: > > We have huge data to sync usually everyday and I wish rsync could guarantee > > performance. > > > > I thought of spliting the directories and run parallel rsyncs on them. It > > may

DO NOT REPLY [Bug 5124] Parallelize the rsync run using multiple threads and/or connections

2009-10-28 Thread samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=5124 m...@mattmccutchen.net changed: What|Removed |Added Summary|Lessons to learn from other |Parallelize the rsync run

Re: OS X 10.6 (Snow Leopard) HFS+ File Compression

2009-10-28 Thread Tony
Thanks. The C code that brkirch provides takes care of a lot of the work, so hopefully someone will be able to provide a patch (Its been over 15 years since I did any C programing, so unfortunately I won't be able to contribute) On Oct 28, 2009, at 12:08 AM, Matt McCutchen wrote: Rsync

Re: OS X 10.6 (Snow Leopard) HFS+ File Compression

2009-10-28 Thread Tony
When rsync 3.0.6 copies files with HFS+ File Compression, the new extended attribute decmpfs is not preserved, and the UF_COMPRESSED flag is not set on the destination and the destination file is not compressed. I examined the destination file as described in ars technica (with ls and

Re: Parallel rsync's for better Performance.

2009-10-28 Thread Matthias Schniedermeyer
On 28.10.2009 09:05, Satish Shukla wrote: > > > Hi , > > We have huge data to sync usually everyday and I wish rsync could guarantee > performance. > > I thought of spliting the directories and run parallel rsyncs on them. It may > cost me some network, but I can control that from the MAX_RS

Parallel rsync's for better Performance.

2009-10-28 Thread Satish Shukla
Hi , We have huge data to sync usually everyday and I wish rsync could guarantee performance. I thought of spliting the directories and run parallel rsyncs on them. It may cost me some network, but I can control that from the MAX_RSYNC_PROCESS variable. Can some one evaluate pros and cons o