Adam Edgar wrote:
It seems the issue is indeed in the ssh layer. scp has the same issue and some
work has been done in “fixing” that:
http://www.psc.edu/index.php/hpn-ssh
From the papers abstract:
Status: O
SCP and the underlying SSH2 protocol implementation in OpenSSH is
network performance limited by statically defined internal flow control
buffers. These buffers often end up acting as a bottleneck for network
throughput of SCP, especially on long and high bandwith network links.
----
It is *A* bottle neck over networks. look for extensions to ssh to
ship unencrypted data streams.
There's a patch for this @ http://www.psc.edu/index.php/hpn-ssh.
However, rsync is dog slow locally as well for exactly the reasons
you mention.
An extract from another note on this topic (came up on suse list this week).
Someone suggested compression for a speed up... I responded to that:
On a local copy or local network, that usually slows down transfers.
[**** 1000:1 speed ratio with large vs. small io sizes):]
One might ask why rsync is so slow --
copying 800G from 1 partition to another via xfsdump/restore takes a bit
under 2 hours,
or about 170MB/s, but with rsync, on the same partition with rsync
transfering
less than 1/1000th as much (700MB), it took ~70-80 minutes... or about
163kB/s.
That's on the same system (local drive -> another local drive)
Transfer speeds depend on many factors. One of the largest is transfer
size (how much
transfered with 1 write /read.
Transfer 1GB, 1-meg at a time, took 2.08s read, and 1.56s to write
(using direct io).
Transfer it at 4K: 37.28s, to read, and 43.02s to write.
So 20-40x can be accounted for just on R/W size (1k buffers were 4x
slower).
Many desktop apps still think 4k is a good "read size"
Over a network, causes drop from 500MB/s down to less than 200KB/s
(as seen in FF and TB) -- 2500X.
Optimal i/o size on my sys is between 16M-256M.
So -- to answer your question, MANY things can affect speed, but I'd
look at the
R/W size first.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
