Re: rsync and many files

2011-06-09 Thread Chris Dunlop
Kevin Korb sanitarium.net> writes: > A lot of this has to do with the filesystems and operating systems involved. > > Since you didn't specify I will guess Linux with ext3. If that is the > case run don't walk to ext4. Also, mount the filesystems with the > noatime and nodiratime options. This

Re: rsync and many files

2011-06-09 Thread Chris Dunlop
On 2011-06-07, Kevin Korb wrote: > > A lot of this has to do with the filesystems and operating systems involved. > > Since you didn't specify I will guess Linux with ext3. If that is the > case run don't walk to ext4. Also, mount the filesystems with the > noatime and nodiratime options. This

AW: rsync and many files

2011-06-07 Thread Cliff Simon
[mailto:k...@sanitarium.net] > Gesendet: Dienstag, 7. Juni 2011 04:52 > An: rsync@lists.samba.org > Betreff: Re: rsync and many files > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > A lot of this has to do with the filesystems and operating systems involved. >

Re: rsync and many files

2011-06-06 Thread Kevin Korb
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 A lot of this has to do with the filesystems and operating systems involved. Since you didn't specify I will guess Linux with ext3. If that is the case run don't walk to ext4. Also, mount the filesystems with the noatime and nodiratime options. Thi

Re: rsync and many files

2011-06-06 Thread Greg Siekas
40 files a second seems very slow. Are you sure the majority of the time is generating the file list and determine what's changed? How many of the millions of files are changed? On modern hardware I see 1000's of files per second when scanning for changed files. On Jun 6, 2011, at 12:39 PM,

Re: rsync and many files

2011-06-06 Thread Steven Levine
In , on 06/06/11 at 12:04 PM, Cliff Simon said: Hi, >We are using rsync via rsnapshot, but this is not elementary. It is used >to backup many (above 100 servers) and works very well. Now there is one >server with many (several millions) files. The files are not very big, so >the complete back

Re: rsync and many files

2011-06-06 Thread Brian K. White
On 6/6/2011 12:54 PM, Paul Slootman wrote: On Mon 06 Jun 2011, Cliff Simon wrote: Hm...I´m using 3.0.3 at the Dest-Server, but now I saw that the Source-Server has 2.6.9 Do I have to enable incremental recursion and from which version is incremental supported? Both ends have to be at least

Re: rsync and many files

2011-06-06 Thread Paul Slootman
On Mon 06 Jun 2011, Cliff Simon wrote: > > Hm...I´m using 3.0.3 at the Dest-Server, but now I saw that the Source-Server > has 2.6.9 > > Do I have to enable incremental recursion and from which version is > incremental supported? Both ends have to be at least 3.0.0 to enable the incremental re

AW: rsync and many files

2011-06-06 Thread Cliff Simon
ootman [mailto:paul+rs...@wurtel.net] > Gesendet: Montag, 6. Juni 2011 18:00 > An: rsync@lists.samba.org > Betreff: Re: rsync and many files > > On Mon 06 Jun 2011, Cliff Simon wrote: > > > > Now my problem is, that the backup needs about 14 hours - the most time > is to gen

Re: rsync and many files

2011-06-06 Thread Paul Slootman
On Mon 06 Jun 2011, Cliff Simon wrote: > > Now my problem is, that the backup needs about 14 hours - the most time is to > generate the filelist and check whether the files are new/changed or not. Are you using a recent version of rsync? One that does incremental recursion? Paul -- Please use

rsync and many files

2011-06-06 Thread Cliff Simon
Hello together, I have a question about using rsync with many files. We are using rsync via rsnapshot, but this is not elementary. It is used to backup many (above 100 servers) and works very well. Now there is one server with many (several millions) files. The files are not very big, so the co