On 8/25/2021 5:29 PM, Takashi Yano wrote:
On Wed, 25 Aug 2021 13:52:19 -0400
Ken Brown wrote:
On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
On Tue, 24 Aug 2021 12:49:52 -0700
Chris Roehrig wrote:
I have a network of Windows, Linux and Mac machines and I use rsync to
synchronize various directories between them.
I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when
the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync
client). In all other scenarios, I get the full 100MB/s as expected from gigabit
ethernet. This has been an ongoing problem for me for a couple of years over
several Windows and Cygwin versions, and I'd like to try to fix it.
If I run rsync --daemon --no-detach under mintty in the foreground on the
remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like
it has something to do with rsync.exe running in the background under the
cygrunsrv+sshd service (which was installed normally using ssh-host-config).
If I do:
pv /dev/zero | ssh $WINHOST "cat > /dev/null"
or even
pv /dev/urandom | ssh $WINHOST md5sum
I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is
being throttled by bandwidth or CPU.
The machines have less than 15% CPU utilization while transferring, with each
of the 4 cores less than 30%, so it doesn't look to be CPU issue.
In Task Manager, sshd.exe and rsync.exe seem to be running normally using only
few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting
their Priority to High doesn't seem to change things.
Looking in Resource Monitor on the remote endpoint, the network usage is pretty
much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me
as if rsync is somehow being bandwidth-throttled when run in the background
under cygsshd.
It's almost as if rsync has an implicit --bwlimit override when it is run from
cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
Any ideas? Not sure where to go from here.
In cygwin, just scp is very slow.
The transfer speed in my environment is as follows.
The tests were done with 100MB of test.dat file.
(1-1) From cygwin-PC,
[yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
yano@linux-server's password:
test.dat 100% 100MB 4.0MB/s 00:24
[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat 100% 100MB 8.0MB/s 00:12
(1-2) From linux-server,
yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
yano@cygwin-PC's password:
test.dat 100% 100MB 4.0MB/s 00:24
yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat 100% 100MB 4.1MB/s 00:24
I looked into this problem, and noticed that this is caused
by cygwin pipe implementation. Pipe in cygwin is configured
with FILE_FLAG_OVERLAPPED.
If the pipe is configured without FILE_FLAG_OVERLAPPED,
the transfer speed is much improved as follows.
(2-1) From cygwin-PC,
[yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
yano@linux-server's password:
test.dat 100% 100MB 85.5MB/s 00:01
[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat 100% 100MB 69.7MB/s 00:01
(2-2) From linux-server,
yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
yano@cygwin-PC's password:
test.dat 100% 100MB 80.1MB/s 00:01
yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat 100% 100MB 57.7MB/s 00:01
I am not sure why this happens and how to fix this.
A couple years ago I had an idea for changing the pipe implementation to avoid
overlapped I/O:
https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
I never followed up on it. But if you think it might help with this problem, I
could dust it off and try to finish it.
Interesting.
It will be also helpfull for:
https://cygwin.com/pipermail/cygwin/2021-March/247987.html
which seems to be the same issue with
https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
However, I wonder why scp dislikes overlapped I/O.
I agree that it would be good to understand this. When I first proposed the
change, I was thinking in terms of code simplification. If it also improves
performance (which we don't know yet), it becomes a higher priority, but in that
case it would be nice to understand why it improves performance.
Ken
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation: https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
