On Mon, 2004-03-22 at 04:42, Hergaarden, Marcel wrote: > We're running rsync 2.5.7 on a Windows2000 server, in combination with > cygwin/ssh. The server who receives the data is a Linux server. > > The amount of data from the Windows server is about 100 Gb. Represented > by 532.000 files of different nature. Mostly doc, ppt and xls files. > > It takes about 2 hours to create only the file list. > > Is the amount of data/files to big, should I segmentate the backupfiles > or is something else the cause of this long duration. >
2 hours for 530,000+ files sounds about right to me. 2 things.. 1) how much does the data change per day? 2) how fast is the network link between the two boxes? We had a situtation recently where a backup via rsync that used to take 1 hour total, suddenly ballooned to 3.5 hours. This wasn't acceptable as it was loading the server down. We had recently upgraded to gigabit ethernet, so we were a bit perplexed. Then we realized that the number of files being rsynced had gone up by a factor of 5. We switched to just doing a simple tar backup of all the files. It only takes an hour again. rsync is great (REALLY GREAT), but remember, its mostly about maximizing bandwidth by only sending whats changed. If 80% of your data changes every day, and you have a gigabit link... rsync isn't really meant for that anyway. This does bring up one point though. Is there any way to optimize file list building? It seems like that turns into a huge bottleneck in the "lots of files" situation. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
