> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Richard Jahnel > > I'vw also tried mbuffer, but I get broken pipe errors part way through > the transfer.
The standard answer is mbuffer. I think you should ask yourself what's going wrong with mbuffer. You're not, by any chance, sending across a LACP aggregated link, are you? Most people don't notice it, but I sure do, that usually LACP introduces packet errors. Just watch your error counter, and start cramming data through there. So far I've never seen a single implementation that passed this test... Although I'm sure I've just had bad luck. If you're having some packet loss, that might explain the poor performance of ssh too. Although ssh is known to slow things down in the best of cases ... I don't know if the speed you're seeing is reasonable considering. > zfs send > pigz > scp arcfour the file gz file to the remote host > gunzip < to zfs receive I would recommend "xz -1" instead of pigz. It's roughly the same speed compression, with approx 2x smaller result. I know xz does multi-threading, but you might just want to "top" it or "prstat" it to be sure. I have a hunch that it can only multi-thread actual files, and maybe can't multi-thread compression of stdin. Maybe. > This takes a very long time for 3 TB of data, and barely makes use the > 10g connection between the machines due to the CPU limiting on the scp > and gunzip processes. That's something ... I know pigz, pbzip2, and xz all compress multi-threaded. But what about uncompress? You say your gunzip is cpu-bound... If the above don't decompress multi-threaded, I know threadzip does multi-thread both compression & decompression. For speed, I would recommend using the "--fast" option. If you consider threadzip, just ignore the comment about "this project is considered dead, in favor of pigz, pbzip2, xz." I'm the guy behind threadzip, and if there's a reason for it to exist, I'll just put that on the page instead. _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
