On Thu, 10 Jul 2008, Glaser, David wrote: > x4500 that we've purchased. Right now I'm using rsync over ssh (via > 1GB/s network) to copy the data but it is almost painfully slow > (700GB over 24 hours). Yeah, it's a load of small files for the most > part. Anyway, would zfs send/receive work better? Do you have to set > up a service on the receiving machine in order to receive the zfs > stream?
You don't need to set up a service on the remote machine. You can use ssh to invoke the zfs receive and pipe the data across the ssh connection, which is similar to what rsync is doing. For example (from the zfs docs): zfs send tank/[EMAIL PROTECTED] | ssh newsys zfs recv sandbox/[EMAIL PROTECTED] For a fresh copy, the bottleneck is quite likely ssh itself. Ssh uses fancy encryption algorithms which take lots of CPU time and really slows things down. The "blowfish" algorithm seems to be fastest so passing -c blowfish As an ssh option can significantly speed things up. For example, this is how you can tell rsync to use ssh with your own options: --rsh='/usr/bin/ssh -c blowfish' In order to achieve even more performance (but without encryption), you can use Netcat as the underlying transport. See http://netcat.sourceforge.net/. Lastly, if you have much more CPU available than bandwidth, then it is worthwhile to install and use the 'lzop' compression program which compresses very quickly to a format only about 30% less compressed than what gzip achieves but fast enough for real-time data transmission. It is easy to insert lzop into the pipeline so that less data is sent across the network. Bob ====================================== Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
