Many thanks for great replies on and off-list.

The suggestions basically ranged from these options:
1. tune TCP on all hosts you wish to transfer between
2. create tuned TCP proxies and transfer through those hosts
3. setup a socat (netcat++) proxy and send through this host
4. use an alternative to plain netcat/scp for large file transfers

My needs are pretty simple: occasionally I need to push large database files (300Gb+) around linux hosts. #4 seems like the best option for me.
People suggested a slew of methods to do this: RBUDP, gridftp, bbcp, and 
many others, with programs either sending with reliable UDP or breaking 
large transfers into multiple streams. Because it was easy to use right 
away, I tried RBUDP and was able to copy a tarball at about 700Mb/s over 
a 20ms delay link, and when factoring in destination disk write speed, 
isn't too bad a starting point. GridFTP and bbcp look very useful too; 
I'll be exploring them as well. The presentation links Kevin O. sent 
were very interesting.
I've looked at HPN-SSH before but haven't played with it much. I'll 
definitely try it out based on the feedback from this thread.
Thanks again.

sk


Sean Knox wrote:
Hi,

I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as "transfer stations" using "reliable UDP" to get around the windowing problem.
I'm thinking of setting up servers with optimized TCP settings to push 
big files around data centers but I'm curious to know how others deal 
with LFN+large transfers.
thanks,
Sean


Reply via email to