poor rsync performance with specific Linux kernel version?

2011-05-17 Thread Michael Doerner - TechnologyWise
Hi all I am new to this list but a happy rsync user for quite some time. Thanks for this great tool. We are experiencing very slow rsync performance when using it as a backup tool for our server virtualisation "Proxmox VE" (http://pve.proxmox.com/wiki/Main_Page). I have searched for our issue an

Re: Multi-Threading?

2011-05-17 Thread Chuck Wolber
On Tue, 17 May 2011, Matt McCutchen wrote: > On Tue, 2011-05-17 at 10:54 -0700, Chuck Wolber wrote: > > > > Abstracting the core functionality into a librsync.so would be really > > spiffy too... > > All easy to say, harder to do (and maintain). I'm thankful that rsync > meets my needs right

Re: Multi-Threading?

2011-05-17 Thread Matt McCutchen
On Tue, 2011-05-17 at 10:54 -0700, Chuck Wolber wrote: > On Tue, 17 May 2011, cac...@quantum-sci.com wrote: > > > In researching this I find that a change to multi-threaded goodness > > would require a massive rewrite, and would only be considered for an > > rsync replacement. > > Abstracting t

Re: Multi-Threading?

2011-05-17 Thread CACook
On Tuesday 17 May, 2011 10:43:21 you wrote: > Wow! That's a lot of data you're transferring into one server, even with a GB > pipe you're using, which appears to be using a fraction of it. Yes, it's my home theater computer with all the movies. I've just set up a RAID array and need to move my

Re: Multi-Threading?

2011-05-17 Thread Chuck Wolber
On Tue, 17 May 2011, cac...@quantum-sci.com wrote: > In researching this I find that a change to multi-threaded goodness > would require a massive rewrite, and would only be considered for an > rsync replacement. Abstracting the core functionality into a librsync.so would be really spiffy too.

Re: Multi-Threading?

2011-05-17 Thread Carlos Carvalho
Matt McCutchen (m...@mattmccutchen.net) wrote on 17 May 2011 12:50: >On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: >> The connexion is Gb enet end-to-end, and is running at only 40Mb/s. >> It has far more capacity than that. The only limiting factor I can >> see is on the ba

Re: Multi-Threading?

2011-05-17 Thread Kevin Korb
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Also, make sure you aren't running with compression as it will just waste CPU cycles. If you are running rsync over ssh it might also be worth while to setup a one time use rsyncd or even NFS mount to eliminate the encryption overhead. On 05/17/11 12

Re: Multi-Threading?

2011-05-17 Thread CACook
Thanks, but the whole function of my backup server pivots on rsync features. Need something rsync-like, but multi-threaded, or with a whole lot less overhead. On Tuesday 17 May, 2011 09:18:01 Chris Hawkins wrote: > I have no idea about potential rsync modifications, but you might try FDT to >

Re: Multi-Threading?

2011-05-17 Thread Matt McCutchen
On Tue, 2011-05-17 at 08:45 -0700, cac...@quantum-sci.com wrote: > The connexion is Gb enet end-to-end, and is running at only 40Mb/s. > It has far more capacity than that. The only limiting factor I can > see is on the backup server one core of the CPU is running 100% rsync. > Clearly rsync is no

Re: Multi-Threading?

2011-05-17 Thread Chris Hawkins
I have no idea about potential rsync modifications, but you might try FDT to solve this problem: http://monalisa.cern.ch/FDT/ I have found it easy to use and it absolutely maxes all available bandwidth for the fastest possible data transfer. It doesn't have rsync goodies like update and only c

Multi-Threading?

2011-05-17 Thread CACook
I have a backup server now restoring 6TB of data to a client machine. This has been going on for four days now, and no sign of getting close to completion. The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It has far more capacity than that. The only limiting factor I can s