(Courtesy copy sent directly to Ben) Eric Whiting <[EMAIL PROTECTED]> wrote: > Ben Ricker wrote: > > > > I am using Rsync between a Redhat Linux box and an AIX RS600. We have > > a about 30gb of database we need to sync to a backup server. Sounds > > good, right? The problem is that Rsync is so slow when we do the > > initial dump. We have files that are 1 - 5gb. It takes around 14-20 > > hours to Rsync the file structure, which means our daily backup will > > miss that days data because we are waiting for the data to rsync. > > ...
I've got 2 areas that I sync across a T-1, as follows: Number of files: 8122 Number of files transferred: 626 Total file size: 2,101,157,198 bytes Total transferred file size: 1,855,602,142 bytes Literal data: 2,851,745 bytes Matched data: 1,852,750,397 bytes File list size: 197,222 Total bytes written: 10,194,719 Total bytes read: 487,766 wrote 10,194,719 bytes read 487,766 bytes 6,841.17 bytes/sec total size is 2,101,157,198 speedup is 196.69 Number of files: 13,502 Number of files transferred: 0 Total file size: Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 304636 Total bytes written: 2741 Total bytes read: 304,713 wrote 2,741 bytes read 304,713 bytes 3,967.15 bytes/sec total size is The above 2 things took 27 minutes 20 seconds total. Remeber again, this is across a T1 (full 1.5 Mbit T1), not 10Mbit or 100Mbit. Without ssh or anything else. > > At first we were using ssh as the agent but we thought that might be > > slowing us down. We then tried to use rsync as a server and push the > > data to the module/zone of the server. Consistently, it takes roughly > > 2 minutes to transfer 10meg of data > > That seems way too long. ... > > I thought it was a network problem so we tried ftp. A 10 mb file took > > 90% the time (about 20 seconds. This is on a 100bit Full Duplex 20 seconds for 10 MB (I'm assuming you meant MegaByte, not MegaBit ;-) is way slow. ESPECIALLY for ftp. A long time ago I ran an ftp test on a 10 Mbit link between 2 Sun Sparc2 machines. I was able to use 99.9% of the bandwidth (i.e. I got over 1 megaByte/sec transfer rates!). Put another way, using 2 sparcstation 2s on a slightly occupied 10Mbit network I should be able to transfer that file in 10 seconds. One word of warning - FTP transfers are way efficient, and can burn all the bandwidth up. Most other protocols are not nearly so efficient, and so you cannot get the speed from them. The same machines and all worked much slower doing rcp as transfer means (but I don't remember the numbers, this was what, 9 years ago?). So, whats the point? Its this: What's your network loading? Have you run anything (like Sun's SE performance toolkit (which unfortunatly only works on sun) to see if your network is working up to snuff? Going off in another direction: Do you have enough RAM? Programs running out of disk instead of out of ram are DEADLY slow... And the amount of data you have to worry about is non-trivial here... ;-) > > switched network with both boxes on the SAME subnet. > > I think you still have other problems. A ftp of a 10Mbyte file takes my > boxes about 1s on a 100Mbit wire. probably due as much to file transfer rates as network speeds! What happens if you do the transfer twice? (So you use the cache for the file data instead of having to read it from the disk - assuming that you're not memory limited!) > Be sure to try ftp both get and put. Oftentimes ... > I think you might have some network issues. I'm going to have to agree. DNS is one possiblility, but that slow ftp reported tends to make me think its something else. > Do a ifconfig on the > linux box to see what sort of errors and collisions you are seeing. And on the AIX box also! What's the loading on that AIX box like? Can you try rsync-ing between 2 linux boxes? Between 2 aix boxes? (First, try ftp transfers and see what kind of performance you get, once THAT is fixed then see how well rsync works between the 'correct' machines, before you worry about trying between other than the machines you plan to use in the final setup) > > Am I missing something? Probably (not much help, eh? ;-) > > Should I expect these speeds? No, this is slow. > > Could there be something on the App switch throttling rsync for some reason? Umm, I think I missed something. What is the app switch? Unless you mean the network switching hub - maybe. Is is SNMP manageable? How much bandwidth is it currently handling? How much CAN it handle? One note on switching hubs - I have one at home, and I've noticed a really weird thing - when transferring data from my laptop (10Mbit PCMCIA on a P133 laptop) to my 'big machine' (Celeron 450MHz with 100Mbit ethernet, lotsa ram, etc) through the switching hub I get collisions(!!!) between the laptop and the hub. Scratch head time, and one of these days I'm going to have to figure out how and why... The point is - don't assume your hub does all its supposed to do and can never have collisions! I think I've deleted the ssh stuff. I had one comment - IF you are bandwidth limited, AND the data compresses nicely, you MIGHT get an improvement using ssh as the transport IF it compresses AND you have spare CPU power for the compress/decompress. (There, did I give enough disclaimers? ;-) rc Rusty E. Carruth Email: [EMAIL PROTECTED] or [EMAIL PROTECTED] Voice: (480) 345-3621 SnailMail: Schlumberger ATE ___ FAX: (480) 345-8793 7855 S. River Parkway, Suite 116 \e/ Ham: N7IKQ @ 146.82+,pl 162.2 Tempe, AZ 85284-1825 V ICBM: 33 20' 44"N 111 53' 47"W http://tuxedo.org/~esr/ecsl/index.html "Why would anyone choose a tool that is the primary virus vector of the known universe?" - me
