On Thu, 26 Aug 2021 18:18:29 -0400 Ken Brown wrote: > On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote: > > On 8/25/2021 5:29 PM, Takashi Yano wrote: > >> On Wed, 25 Aug 2021 13:52:19 -0400 > >> Ken Brown wrote: > >>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: > >>>> On Tue, 24 Aug 2021 12:49:52 -0700 > >>>> Chris Roehrig wrote: > >>>>> I have a network of Windows, Linux and Mac machines and I use rsync to > >>>>> synchronize various directories between them. > >>>>> > >>>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) > >>>>> only > >>>>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux > >>>>> or > >>>>> Cygwin rsync client). In all other scenarios, I get the full 100MB/s > >>>>> as > >>>>> expected from gigabit ethernet. This has been an ongoing problem for > >>>>> me > >>>>> for a couple of years over several Windows and Cygwin versions, and I'd > >>>>> like to try to fix it. > >>>>> > >>>>> If I run rsync --daemon --no-detach under mintty in the foreground on > >>>>> the > >>>>> remote Windows endpoint, I get the full 100 MB/s transfers, so it > >>>>> seems > >>>>> like it has something to do with rsync.exe running in the background > >>>>> under > >>>>> the cygrunsrv+sshd service (which was installed normally using > >>>>> ssh-host-config). > >>>>> > >>>>> If I do: > >>>>> pv /dev/zero | ssh $WINHOST "cat > /dev/null" > >>>>> or even > >>>>> pv /dev/urandom | ssh $WINHOST md5sum > >>>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd > >>>>> itself > >>>>> is being throttled by bandwidth or CPU. > >>>>> > >>>>> The machines have less than 15% CPU utilization while transferring, > >>>>> with > >>>>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue. > >>>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally > >>>>> using > >>>>> only few percent CPU, and show Power Throttling=Disabled, > >>>>> Priority=Normal. Setting their Priority to High doesn't seem to > >>>>> change > >>>>> things. > >>>>> > >>>>> Looking in Resource Monitor on the remote endpoint, the network usage > >>>>> is > >>>>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it > >>>>> sure > >>>>> looks to me as if rsync is somehow being bandwidth-throttled when run > >>>>> in > >>>>> the background under cygsshd. > >>>>> > >>>>> It's almost as if rsync has an implicit --bwlimit override when it is > >>>>> run > >>>>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes > >>>>> no > >>>>> difference). > >>>>> > >>>>> > >>>>> Any ideas? Not sure where to go from here. > >>>> > >>>> In cygwin, just scp is very slow. > >>>> > >>>> The transfer speed in my environment is as follows. > >>>> The tests were done with 100MB of test.dat file. > >>>> > >>>> (1-1) From cygwin-PC, > >>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > >>>> yano@linux-server's password: > >>>> test.dat 100% 100MB 4.0MB/s > >>>> 00:24 > >>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > >>>> yano@linux-server's password: > >>>> test.dat 100% 100MB 8.0MB/s > >>>> 00:12 > >>>> > >>>> (1-2) From linux-server, > >>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > >>>> yano@cygwin-PC's password: > >>>> test.dat 100% 100MB 4.0MB/s > >>>> 00:24 > >>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > >>>> yano@cygwin-PC's password: > >>>> test.dat 100% 100MB 4.1MB/s > >>>> 00:24 > >>>> > >>>> > >>>> I looked into this problem, and noticed that this is caused > >>>> by cygwin pipe implementation. Pipe in cygwin is configured > >>>> with FILE_FLAG_OVERLAPPED. > >>>> > >>>> If the pipe is configured without FILE_FLAG_OVERLAPPED, > >>>> the transfer speed is much improved as follows. > >>>> > >>>> > >>>> (2-1) From cygwin-PC, > >>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. > >>>> yano@linux-server's password: > >>>> test.dat 100% 100MB 85.5MB/s > >>>> 00:01 > >>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . > >>>> yano@linux-server's password: > >>>> test.dat 100% 100MB 69.7MB/s > >>>> 00:01 > >>>> > >>>> (2-2) From linux-server, > >>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . > >>>> yano@cygwin-PC's password: > >>>> test.dat 100% 100MB 80.1MB/s > >>>> 00:01 > >>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. > >>>> yano@cygwin-PC's password: > >>>> test.dat 100% 100MB 57.7MB/s > >>>> 00:01 > >>>> > >>>> I am not sure why this happens and how to fix this. > >>> > >>> A couple years ago I had an idea for changing the pipe implementation to > >>> avoid > >>> overlapped I/O: > >>> > >>> https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html > >>> https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html > >>> > >>> I never followed up on it. But if you think it might help with this > >>> problem, I > >>> could dust it off and try to finish it. > >> > >> Interesting. > >> > >> It will be also helpfull for: > >> https://cygwin.com/pipermail/cygwin/2021-March/247987.html > >> which seems to be the same issue with > >> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app > >> > >> > >> > >> However, I wonder why scp dislikes overlapped I/O. > > > > I agree that it would be good to understand this. When I first proposed > > the > > change, I was thinking in terms of code simplification. If it also > > improves > > performance (which we don't know yet), it becomes a higher priority, but in > > that > > case it would be nice to understand why it improves performance. > > Hi Takashi, > > In case you want to try out my proposed change, I've just rebased the patches > to > the current master and pushed them to a new topic/pipe branch.
Hi Ken, Thanks much! I tested topic/pipe branch. [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. yano@linux-server's password: test.dat 100% 100MB 95.9MB/s 00:01 [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 8.0MB/s 00:12 yano@linux-server:~$ scp yano@cygwin-PC:test.dat . yano@cygwin-PC's password: test.dat 100% 100MB 109.7MB/s 00:00 yano@linux-server:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 31.4MB/s 00:03 As shown above, outgoing transfer-rate has been improved upto near theoretical limit. However, incoming transfer-rate is not improved much. I digged further and found the first patch attached solves the issue as follows. [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . yano@linux-server's password: test.dat 100% 100MB 112.8MB/s 00:00 yano@linux-server2:~$ scp test.dat yano@cygwin-PC:. yano@cygwin-PC's password: test.dat 100% 100MB 102.5MB/s 00:00 I also tested the case: > >> https://cygwin.com/pipermail/cygwin/2021-March/247987.html > >> which seems to be the same issue with > >> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app Unfortunately, topic/pipe does not help. I confirmed that applying the second patch attached, which reverts to create() rather than nt_create(), and setting CYGWIN=pipe_byte fixes the problem. What do you think of this alternative implementation which does not use nt_create()? -- Takashi Yano <takashi.y...@nifty.ne.jp>
0001-Cygwin-select-Improve-select-poll-response.patch
Description: Binary data
0002-Cygwin-pipe-Revert-to-create-rather-than-nt_create.patch
Description: Binary data
-- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation: https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
