Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano via Cygwin]

Hi Ken,

On Sat, 28 Aug 2021 16:55:52 -0400
Ken Brown wrote:
> On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 13:58:08 +0200
> > Corinna Vinschen wrote:
> >> On Aug 28 18:41, Takashi Yano via Cygwin wrote:
> >>> On Sat, 28 Aug 2021 10:43:27 +0200
> >>> Corinna Vinschen wrote:
>  On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > On Fri, 27 Aug 2021 12:00:50 -0400
> > Ken Brown wrote:
> >> Two years ago I thought I needed nt_create to avoid problems when 
> >> calling
> >> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> >> set_pipe_non_blocking unnecessary?  Is that the point of your 
> >> modification to
> >> raw_read?
> >
> > Yes. Instead of making windows read function itself non-blocking,
> > it is possible to check if the pipe can be read before read using
> > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > returned.
> 
>  The problem is this:
> 
> if (PeekNamedPipe())
>   ReadFile(blocking);
> 
>  is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>  thread from draining the pipe between the PeekNamedPipe and the ReadFile
>  call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>  stop it via a signal.
> >>>
> >>> Hmm, you are right. Mutex guard seems to be necessary like pty code
> >>> if we go this way.
> >>>
>  Is a blocking ReadFile actually faster than a non-blocking read?  Or
>  does it mainly depend on BYTE vs. MESSAGE mode?
> >>>
> >>> Actually, I don't think so. Perhaps it is not essential problem of
> >>> overlapped I/O but something is wrong with current pipe code.
> >>>
>  What if the pipe is created non-blocking and stays non-blocking all the
>  time and uses BYTE mode all the time?  Just as sockets, it would always
>  only emulate blocking mode.  Wouldn't that drop code size a lot and fix
>  most problems?
> >>>
> >>> If 'non-blocking' means overlapped I/O, only the problem will be:
> >>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> >>
> >> Sorry if that wasn't clear, but I was not talking about overlapped I/O,
> >> which we should get rid off, but of real non-blocking mode, which
> >> Windows pipes are fortunately capable of.
> > 
> > Do you mean, PIPE_NOWAIT flag? If this flags is specified in
> > the read pipe, non-cygwin apps cannot read the pipe correctly.
> 
> While waiting for Corinna's response to this, I have one more question.  Do 
> you 
> understand why nt_create() failed and you had to revert to create()?  Was it 
> an 
> access problem because nt_create requested FILE_WRITE_ATTRIBUTES?  Or did I 
> make 
> some careless mistake in writing nt_create?

I am sorry but no. I don't understand why piping C# program via
the pipe created by nt_create() has the issue. I tried to change
setup parameters in nt_create(), however, I did not succeed it to
work. I also couldn't find any mistake in nt_create() so far.

Win32 programs which use ReadFile() and WriteFile() work even
with the pipe created by nt_create() as well as overlapped I/O.

What does C# program differ from legacy win32 program at all?

-- 
Takashi Yano 

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano via Cygwin]

On Sat, 28 Aug 2021 18:41:02 +0900
Takashi Yano wrote:
> On Sat, 28 Aug 2021 10:43:27 +0200
> Corinna Vinschen wrote:
> > On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > > On Fri, 27 Aug 2021 12:00:50 -0400
> > > Ken Brown wrote:
> > > > Two years ago I thought I needed nt_create to avoid problems when 
> > > > calling 
> > > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > > set_pipe_non_blocking unnecessary?  Is that the point of your 
> > > > modification to 
> > > > raw_read?
> > > 
> > > Yes. Instead of making windows read function itself non-blocking,
> > > it is possible to check if the pipe can be read before read using
> > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > > returned.
> > 
> > The problem is this:
> > 
> >   if (PeekNamedPipe())
> > ReadFile(blocking);
> > 
> > is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > stop it via a signal.
> 
> Hmm, you are right. Mutex guard seems to be necessary like pty code
> if we go this way.

I have found that set_pipe_non_blocking() succeeds for both read and
write pipes if the write pipe is created by CreateNamedPipe() and the
read pipe is created by CreateFile() contrary to the current create()
code. Therefore, not only nt_create() but also PeekNamedPipe() become
unnecessary.

Please see the revised patch attached.

-- 
Takashi Yano 


v2-0002-Cygwin-pipe-Revert-to-create-rather-than-nt_creat.patch
Description: Binary data

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Corinna Vinschen via Cygwin]

On Aug 29 00:43, Takashi Yano via Cygwin wrote:
> On Sat, 28 Aug 2021 13:58:08 +0200
> Corinna Vinschen wrote:
> > On Aug 28 18:41, Takashi Yano via Cygwin wrote:
> > > On Sat, 28 Aug 2021 10:43:27 +0200
> > > Corinna Vinschen wrote:
> > > > On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > > > > On Fri, 27 Aug 2021 12:00:50 -0400
> > > > > Ken Brown wrote:
> > > > > > Two years ago I thought I needed nt_create to avoid problems when 
> > > > > > calling 
> > > > > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > > > > set_pipe_non_blocking unnecessary?  Is that the point of your 
> > > > > > modification to 
> > > > > > raw_read?
> > > > > 
> > > > > Yes. Instead of making windows read function itself non-blocking,
> > > > > it is possible to check if the pipe can be read before read using
> > > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > > > > returned.
> > > > 
> > > > The problem is this:
> > > > 
> > > >   if (PeekNamedPipe())
> > > > ReadFile(blocking);
> > > > 
> > > > is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > > > thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > > > call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > > > stop it via a signal.
> > > 
> > > Hmm, you are right. Mutex guard seems to be necessary like pty code
> > > if we go this way.
> > > 
> > > > Is a blocking ReadFile actually faster than a non-blocking read?  Or
> > > > does it mainly depend on BYTE vs. MESSAGE mode?
> > > 
> > > Actually, I don't think so. Perhaps it is not essential problem of
> > > overlapped I/O but something is wrong with current pipe code.
> > > 
> > > > What if the pipe is created non-blocking and stays non-blocking all the
> > > > time and uses BYTE mode all the time?  Just as sockets, it would always
> > > > only emulate blocking mode.  Wouldn't that drop code size a lot and fix
> > > > most problems?
> > > 
> > > If 'non-blocking' means overlapped I/O, only the problem will be:
> > > https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> > 
> > Sorry if that wasn't clear, but I was not talking about overlapped I/O,
> > which we should get rid off, but of real non-blocking mode, which
> > Windows pipes are fortunately capable of.
> 
> Do you mean, PIPE_NOWAIT flag? If this flags is specified in
> the read pipe, non-cygwin apps cannot read the pipe correctly.

Do you mean after execve?  If the child is a non-Cygwin process, we
could run a loop over the stdio descriptors in child_info_spawn::worker
setting pipes to blocking.

But that's a corner problem, easily solved.  It's more important that it
works nicely for Cygwin processes.


Corinna

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: updatedb broken as of findutils 4.8.0-1 due to bigram.exe no longer being provided

[Hans-Bernhard Bröker]

Am 28.08.2021 um 18:23 schrieb Dan Harkless:

Looks like it's because in findutils 4.8.0-1, the bigram.exe program is 
no longer provided, but the /usr/bin/updatedb script (still) depends on 
it being there:





     [...]
     + for binary in $find $frcode $bigram $code
     + checkbinary /usr/libexec/frcode


The version of updatedb in the 4.8.0-1 package does not actually contain 
those lines.  Mention of both $bigram and $code has been removed from 
the loop construct (and from everywhere else in the script).


That's because the old format of find databases, which is the only one 
actually using bigram and code, was removed from updatedb as of 
findutils version 4.7, so there really cannot be a need for the bigram 
tool any more.



--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: updatedb broken as of findutils 4.8.0-1 due to bigram.exe no longer being provided

[Dan Harkless]

On 8/29/2021 4:02 AM, Hans-Bernhard Bröker wrote:

Am 28.08.2021 um 18:23 schrieb Dan Harkless:
Looks like it's because in findutils 4.8.0-1, the bigram.exe program 
is no longer provided, but the /usr/bin/updatedb script (still) 
depends on it being there:

 [...]

 + for binary in $find $frcode $bigram $code
 + checkbinary /usr/libexec/frcode


The version of updatedb in the 4.8.0-1 package does not actually 
contain those lines.  Mention of both $bigram and $code has been 
removed from the loop construct (and from everywhere else in the script).


That's because the old format of find databases, which is the only one 
actually using bigram and code, was removed from updatedb as of 
findutils version 4.7, so there really cannot be a need for the bigram 
tool any more.


Argh!  So sorry for the false report!  I completely forgot that years 
back I had made a locally patched version (which is earlier in my path) 
of Cygwin updatedb 4.6.0-1 to troubleshoot and work around problems I 
was having with the tool.


I have 12M+ pathnames on my main Windows system, and I suddenly started 
having issues with the updatedb going from taking less than an hour, to 
taking more than 24 hours, and running into the next job.


It was very awkward to try to troubleshoot what was going on without a 
'find' log to 'tail', so I patched my  local copy of updatedb to write 
to an intermediate file, rather than going direct to 'sort' over a pipe.


Another problem I was having was that though I have 24 GB of RAM on my 
system, I would get low-memory popup warnings from the OS when the sort 
would go off.  (The warnings mislay the blame on Firefox, because I 
usually have big sessions running that take even more RAM than the sort.)


I was hoping running sort on a _file_ rather than stdin might allow it 
to reduce the RAM use enough to not get the warning, but unfortunately 
(and unsurprisingly) I still get it with the intermediate file.  This is 
just a warning, though — I haven't had it actually run out of RAM so 
far, I don't think.


The final problem I was addressing in my patched version was some 
missing error-checking, which was causing me to be left with _no_ 
filename DB, when the update would fail, rather than at least being left 
with the one from last time.


I could send along my patches, but I don't know that I've solved these 
issues in a general enough way.  For instance, my 12 million+ pathnames 
come out to about 1.4 GiB of UNIX-linefeed-separated UTF-8 strings.  
Writing that much to my HD is not a concern, but obviously some people 
might not want to write that much every time to, say, a small 
flash-based device.


Thoughts?

--
Dan Harkless



--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown via Cygwin]

On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:

On Sat, 28 Aug 2021 18:41:02 +0900
Takashi Yano wrote:

On Sat, 28 Aug 2021 10:43:27 +0200
Corinna Vinschen wrote:

On Aug 28 02:21, Takashi Yano via Cygwin wrote:

On Fri, 27 Aug 2021 12:00:50 -0400
Ken Brown wrote:

Two years ago I thought I needed nt_create to avoid problems when calling
set_pipe_non_blocking.  Are you saying that's not an issue?  Is
set_pipe_non_blocking unnecessary?  Is that the point of your modification to
raw_read?


Yes. Instead of making windows read function itself non-blocking,
it is possible to check if the pipe can be read before read using
PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
returned.


The problem is this:

   if (PeekNamedPipe())
 ReadFile(blocking);

is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
thread from draining the pipe between the PeekNamedPipe and the ReadFile
call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
stop it via a signal.


Hmm, you are right. Mutex guard seems to be necessary like pty code
if we go this way.


I have found that set_pipe_non_blocking() succeeds for both read and
write pipes if the write pipe is created by CreateNamedPipe() and the
read pipe is created by CreateFile() contrary to the current create()
code. Therefore, not only nt_create() but also PeekNamedPipe() become
unnecessary.

Please see the revised patch attached.


That's a great idea.

I've applied your two patches to the topic/pipe branch.  I also rebased it and 
did a forced push in order to bring in Corinna's loader script fix.  So you'll 
have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'.


Does this now fix all known problems with pipes?

Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT, it 
wouldn't decrease the code size at this point, so the only question is whether 
it might improve performance.


If you think it's worth trying, I'd be glad to code it up on a new branch, and 
we could compare the two.


Aside from that, I'm wondering how and when to merge the new pipe implementation 
to master.  It obviously needs much more widespread testing than it's gotten so 
far.  I'm a little nervous about it because I haven't thought about the details 
for two years, and no one other than me has tested it until a few days ago.


Ken

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

I'd be happy to test it out if you can give me some commands to git it and 
build it to replace my existing stock Cygwin installation.
(Or it is just the cygwin.dll I'd need to replace?)

My daily backup scripts use a lot of pipes and named pipes.

-- Chris



On Sun Aug 29 2021, at 8:57 AM, Ken Brown via Cygwin  wrote:
> 
> Aside from that, I'm wondering how and when to merge the new pipe 
> implementation to master.  It obviously needs much more widespread testing 
> than it's gotten so far.  I'm a little nervous about it because I haven't 
> thought about the details for two years, and no one other than me has tested 
> it until a few days ago.
> 
> Ken
> 
> -- 
> Problem reports:  https://cygwin.com/problems.html
> FAQ:  https://cygwin.com/faq/
> Documentation:https://cygwin.com/docs.html
> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano via Cygwin]

On Sun, 29 Aug 2021 11:57:04 -0400
Ken Brown wrote:
> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 18:41:02 +0900
> > Takashi Yano wrote:
> >> On Sat, 28 Aug 2021 10:43:27 +0200
> >> Corinna Vinschen wrote:
> >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>  On Fri, 27 Aug 2021 12:00:50 -0400
>  Ken Brown wrote:
> > Two years ago I thought I needed nt_create to avoid problems when 
> > calling
> > set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> > set_pipe_non_blocking unnecessary?  Is that the point of your 
> > modification to
> > raw_read?
> 
>  Yes. Instead of making windows read function itself non-blocking,
>  it is possible to check if the pipe can be read before read using
>  PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>  returned.
> >>>
> >>> The problem is this:
> >>>
> >>>if (PeekNamedPipe())
> >>>  ReadFile(blocking);
> >>>
> >>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> >>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> >>> stop it via a signal.
> >>
> >> Hmm, you are right. Mutex guard seems to be necessary like pty code
> >> if we go this way.
> > 
> > I have found that set_pipe_non_blocking() succeeds for both read and
> > write pipes if the write pipe is created by CreateNamedPipe() and the
> > read pipe is created by CreateFile() contrary to the current create()
> > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > unnecessary.
> > 
> > Please see the revised patch attached.
> 
> That's a great idea.
> 
> I've applied your two patches to the topic/pipe branch.  I also rebased it 
> and 
> did a forced push in order to bring in Corinna's loader script fix.  So 
> you'll 
> have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'.
> 
> Does this now fix all known problems with pipes?

Only the small thing remaining is pipe mode. If the pipe mode
is changed to byte mode, the following issue will be solved.
https://cygwin.com/pipermail/cygwin/2021-March/247987.html

How about the simple patch attached?

The comment in pipe code says:
 Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
 This *seems* to more closely mimic Linux pipe behavior and is
 definitely required for pty handling since fhandler_pty_master
 writes to the pipe in chunks, terminated by newline when CANON mode
 is specified.

This mentions about pty behaiviour in canonical mode, however, the
pty pipe is created as message mode even with this patch. Are there
any other reasons that message mode is preferred for pipe?

-- 
Takashi Yano 


0003-Cygwin-pipe-Set-byte-mode-as-default-rather-than-mes.patch
Description: Binary data

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown via Cygwin]

On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:

On Sat, 28 Aug 2021 18:41:02 +0900
Takashi Yano wrote:

On Sat, 28 Aug 2021 10:43:27 +0200
Corinna Vinschen wrote:

On Aug 28 02:21, Takashi Yano via Cygwin wrote:

On Fri, 27 Aug 2021 12:00:50 -0400
Ken Brown wrote:

Two years ago I thought I needed nt_create to avoid problems when calling
set_pipe_non_blocking.  Are you saying that's not an issue?  Is
set_pipe_non_blocking unnecessary?  Is that the point of your modification to
raw_read?


Yes. Instead of making windows read function itself non-blocking,
it is possible to check if the pipe can be read before read using
PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
returned.


The problem is this:

   if (PeekNamedPipe())
 ReadFile(blocking);

is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
thread from draining the pipe between the PeekNamedPipe and the ReadFile
call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
stop it via a signal.


Hmm, you are right. Mutex guard seems to be necessary like pty code
if we go this way.


I have found that set_pipe_non_blocking() succeeds for both read and
write pipes if the write pipe is created by CreateNamedPipe() and the
read pipe is created by CreateFile() contrary to the current create()
code. Therefore, not only nt_create() but also PeekNamedPipe() become
unnecessary.

Please see the revised patch attached.


I haven't had a chance to test this myself yet, but occurs to me that we might 
have a different problem after this patch: Does the write handle that we get 
from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?


Ken

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown via Cygwin]

On 8/29/2021 3:37 PM, Takashi Yano wrote:

On Sun, 29 Aug 2021 11:57:04 -0400
Ken Brown wrote:

On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:

On Sat, 28 Aug 2021 18:41:02 +0900
Takashi Yano wrote:

On Sat, 28 Aug 2021 10:43:27 +0200
Corinna Vinschen wrote:

On Aug 28 02:21, Takashi Yano via Cygwin wrote:

On Fri, 27 Aug 2021 12:00:50 -0400
Ken Brown wrote:

Two years ago I thought I needed nt_create to avoid problems when calling
set_pipe_non_blocking.  Are you saying that's not an issue?  Is
set_pipe_non_blocking unnecessary?  Is that the point of your modification to
raw_read?


Yes. Instead of making windows read function itself non-blocking,
it is possible to check if the pipe can be read before read using
PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
returned.


The problem is this:

if (PeekNamedPipe())
  ReadFile(blocking);

is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
thread from draining the pipe between the PeekNamedPipe and the ReadFile
call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
stop it via a signal.


Hmm, you are right. Mutex guard seems to be necessary like pty code
if we go this way.


I have found that set_pipe_non_blocking() succeeds for both read and
write pipes if the write pipe is created by CreateNamedPipe() and the
read pipe is created by CreateFile() contrary to the current create()
code. Therefore, not only nt_create() but also PeekNamedPipe() become
unnecessary.

Please see the revised patch attached.


That's a great idea.

I've applied your two patches to the topic/pipe branch.  I also rebased it and
did a forced push in order to bring in Corinna's loader script fix.  So you'll
have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'.

Does this now fix all known problems with pipes?


Only the small thing remaining is pipe mode. If the pipe mode
is changed to byte mode, the following issue will be solved.
https://cygwin.com/pipermail/cygwin/2021-March/247987.html

How about the simple patch attached?

The comment in pipe code says:
  Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
  This *seems* to more closely mimic Linux pipe behavior and is
  definitely required for pty handling since fhandler_pty_master
  writes to the pipe in chunks, terminated by newline when CANON mode
  is specified.

This mentions about pty behaiviour in canonical mode, however, the
pty pipe is created as message mode even with this patch. Are there
any other reasons that message mode is preferred for pipe?


No idea.  All I remember is that there was a lot of discussion around the time 
that it was decided to use PIPE_TYPE_MESSAGE by default.  Corinna probably 
remembers the reasons.


Ken

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown via Cygwin]

On 8/29/2021 3:24 PM, Chris Roehrig wrote:

I'd be happy to test it out if you can give me some commands to git it and 
build it to replace my existing stock Cygwin installation.
(Or it is just the cygwin.dll I'd need to replace?)

My daily backup scripts use a lot of pipes and named pipes.


It's just cygwin1.dll that you'd need to replace.  Instructions for building the 
latter can be found at


  https://www.cygwin.com/faq.html#faq.programming.building-cygwin

but they're out of date in two respects:

1. Currently the build does *not* ignore errors in building the documentation.

2. In addition to the packages listed in the FAQ, you'll need the following:

  texlive-collection-latexrecommended
  texlive-collection-fontsrecommended
  texlive-collection-pictures
  python38-lxml
  python38-ply

These instructions will produce cygwin1.dll in the bin subdirectory of your 
install directory, built from the git master branch.


If you want to try the topic/pipe branch, just go to the newlib-cygwin git repo 
that you cloned, issue the command 'git checkout topic/pipe', and rebuild.


I'll be glad to help if you run into problems.

Ken

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown via Cygwin]

On 8/29/2021 4:41 AM, Takashi Yano wrote:

Hi Ken,

On Sat, 28 Aug 2021 16:55:52 -0400
Ken Brown wrote:

On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote:

On Sat, 28 Aug 2021 13:58:08 +0200
Corinna Vinschen wrote:

On Aug 28 18:41, Takashi Yano via Cygwin wrote:

On Sat, 28 Aug 2021 10:43:27 +0200
Corinna Vinschen wrote:

On Aug 28 02:21, Takashi Yano via Cygwin wrote:

On Fri, 27 Aug 2021 12:00:50 -0400
Ken Brown wrote:

Two years ago I thought I needed nt_create to avoid problems when calling
set_pipe_non_blocking.  Are you saying that's not an issue?  Is
set_pipe_non_blocking unnecessary?  Is that the point of your modification to
raw_read?


Yes. Instead of making windows read function itself non-blocking,
it is possible to check if the pipe can be read before read using
PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
returned.


The problem is this:

if (PeekNamedPipe())
  ReadFile(blocking);

is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
thread from draining the pipe between the PeekNamedPipe and the ReadFile
call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
stop it via a signal.


Hmm, you are right. Mutex guard seems to be necessary like pty code
if we go this way.


Is a blocking ReadFile actually faster than a non-blocking read?  Or
does it mainly depend on BYTE vs. MESSAGE mode?


Actually, I don't think so. Perhaps it is not essential problem of
overlapped I/O but something is wrong with current pipe code.


What if the pipe is created non-blocking and stays non-blocking all the
time and uses BYTE mode all the time?  Just as sockets, it would always
only emulate blocking mode.  Wouldn't that drop code size a lot and fix
most problems?


If 'non-blocking' means overlapped I/O, only the problem will be:
https://cygwin.com/pipermail/cygwin/2021-March/247987.html


Sorry if that wasn't clear, but I was not talking about overlapped I/O,
which we should get rid off, but of real non-blocking mode, which
Windows pipes are fortunately capable of.


Do you mean, PIPE_NOWAIT flag? If this flags is specified in
the read pipe, non-cygwin apps cannot read the pipe correctly.


While waiting for Corinna's response to this, I have one more question.  Do you
understand why nt_create() failed and you had to revert to create()?  Was it an
access problem because nt_create requested FILE_WRITE_ATTRIBUTES?  Or did I make
some careless mistake in writing nt_create?


I am sorry but no. I don't understand why piping C# program via
the pipe created by nt_create() has the issue. I tried to change
setup parameters in nt_create(), however, I did not succeed it to
work. I also couldn't find any mistake in nt_create() so far.

Win32 programs which use ReadFile() and WriteFile() work even
with the pipe created by nt_create() as well as overlapped I/O.

What does C# program differ from legacy win32 program at all?


I don't know.

By the way, when I introduced nt_create(), my preference would have been to 
simply change create() to use the NT API, but I was afraid to do that because I 
didn't want to take a chance on breaking something.  That's still my preference, 
if we can find a way to work around this problem with C# programs.


Ken

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: updatedb broken as of findutils 4.8.0-1 due to bigram.exe no longer being provided

[Brian Inglis]

On 2021-08-29 06:06, Dan Harkless wrote:

On 8/29/2021 4:02 AM, Hans-Bernhard Bröker wrote:

Am 28.08.2021 um 18:23 schrieb Dan Harkless:
Looks like it's because in findutils 4.8.0-1, the bigram.exe program 
is no longer provided, but the /usr/bin/updatedb script (still) 
depends on it being there:

 [...]

 + for binary in $find $frcode $bigram $code
 + checkbinary /usr/libexec/frcode


The version of updatedb in the 4.8.0-1 package does not actually 
contain those lines.  Mention of both $bigram and $code has been 
removed from the loop construct (and from everywhere else in the script).


That's because the old format of find databases, which is the only one 
actually using bigram and code, was removed from updatedb as of 
findutils version 4.7, so there really cannot be a need for the bigram 
tool any more.


Argh!  So sorry for the false report!  I completely forgot that years 
back I had made a locally patched version (which is earlier in my path) 
of Cygwin updatedb 4.6.0-1 to troubleshoot and work around problems I 
was having with the tool.


I have 12M+ pathnames on my main Windows system, and I suddenly started 
having issues with the updatedb going from taking less than an hour, to 
taking more than 24 hours, and running into the next job.


It was very awkward to try to troubleshoot what was going on without a 
'find' log to 'tail', so I patched my  local copy of updatedb to write 
to an intermediate file, rather than going direct to 'sort' over a pipe.


Another problem I was having was that though I have 24 GB of RAM on my 
system, I would get low-memory popup warnings from the OS when the sort 
would go off.  (The warnings mislay the blame on Firefox, because I 
usually have big sessions running that take even more RAM than the sort.)


I was hoping running sort on a _file_ rather than stdin might allow it 
to reduce the RAM use enough to not get the warning, but unfortunately 
(and unsurprisingly) I still get it with the intermediate file.  This is 
just a warning, though — I haven't had it actually run out of RAM so 
far, I don't think.


The final problem I was addressing in my patched version was some 
missing error-checking, which was causing me to be left with _no_ 
filename DB, when the update would fail, rather than at least being left 
with the one from last time.


I could send along my patches, but I don't know that I've solved these 
issues in a general enough way.  For instance, my 12 million+ pathnames 
come out to about 1.4 GiB of UNIX-linefeed-separated UTF-8 strings. 
Writing that much to my HD is not a concern, but obviously some people 
might not want to write that much every time to, say, a small 
flash-based device.


Thoughts?


Thanks for the analysis Hans-Bernhard.

Please recheck the announcement for 4.8 and change info for 4.7: as of 
4.8 locate should still work on old format dbs, but from 4.7 updatedb 
will no longer generate or update them, and in some future release, 
locate will no longer work on them.
The old (pre-GNU Unix) format was deprecated from 4.0 (~25 years ago!) 
and each run of updatedb should have warned you to upgrade, unless you 
patched that out.


See:

$ info finding databases 'database formats' old

or:



I searched for more info on the discussion list archive at:



but could find nothing obviously related to upgrading or migrating, 
although that archive goes back only ~20 years! ;^>


Migration appears to require running the previous 4.6 updatedb without 
--old-format to regenerate the new database in LOCATE02 format?
You should then be able to upgrade to the latest 4.8 findutils and use 
that going forward.


You could email the discussion list  about 
your situation, file sizes, timings, migration path, and issues, and 
cross-post here about anything in the replies we may be able to help you 
with.


--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano via Cygwin]

On Sun, 29 Aug 2021 17:04:56 -0400
Ken Brown wrote:
> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 18:41:02 +0900
> > Takashi Yano wrote:
> >> On Sat, 28 Aug 2021 10:43:27 +0200
> >> Corinna Vinschen wrote:
> >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>  On Fri, 27 Aug 2021 12:00:50 -0400
>  Ken Brown wrote:
> > Two years ago I thought I needed nt_create to avoid problems when 
> > calling
> > set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> > set_pipe_non_blocking unnecessary?  Is that the point of your 
> > modification to
> > raw_read?
> 
>  Yes. Instead of making windows read function itself non-blocking,
>  it is possible to check if the pipe can be read before read using
>  PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>  returned.
> >>>
> >>> The problem is this:
> >>>
> >>>if (PeekNamedPipe())
> >>>  ReadFile(blocking);
> >>>
> >>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> >>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> >>> stop it via a signal.
> >>
> >> Hmm, you are right. Mutex guard seems to be necessary like pty code
> >> if we go this way.
> > 
> > I have found that set_pipe_non_blocking() succeeds for both read and
> > write pipes if the write pipe is created by CreateNamedPipe() and the
> > read pipe is created by CreateFile() contrary to the current create()
> > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > unnecessary.
> > 
> > Please see the revised patch attached.
> 
> I haven't had a chance to test this myself yet, but occurs to me that we 
> might 
> have a different problem after this patch: Does the write handle that we get 
> from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?

I have just checked this, and the answer is "No". Due to this problem,
NtQueryInformationFile() call in select() fails on the write pipe.

It seems that we need more consideration...

-- 
Takashi Yano 

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano via Cygwin]

On Mon, 30 Aug 2021 09:13:14 +0900
Takashi Yano wrote:
> On Sun, 29 Aug 2021 17:04:56 -0400
> Ken Brown wrote:
> > On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > > On Sat, 28 Aug 2021 18:41:02 +0900
> > > Takashi Yano wrote:
> > >> On Sat, 28 Aug 2021 10:43:27 +0200
> > >> Corinna Vinschen wrote:
> > >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> >  On Fri, 27 Aug 2021 12:00:50 -0400
> >  Ken Brown wrote:
> > > Two years ago I thought I needed nt_create to avoid problems when 
> > > calling
> > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> > > set_pipe_non_blocking unnecessary?  Is that the point of your 
> > > modification to
> > > raw_read?
> > 
> >  Yes. Instead of making windows read function itself non-blocking,
> >  it is possible to check if the pipe can be read before read using
> >  PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> >  returned.
> > >>>
> > >>> The problem is this:
> > >>>
> > >>>if (PeekNamedPipe())
> > >>>  ReadFile(blocking);
> > >>>
> > >>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > >>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > >>> stop it via a signal.
> > >>
> > >> Hmm, you are right. Mutex guard seems to be necessary like pty code
> > >> if we go this way.
> > > 
> > > I have found that set_pipe_non_blocking() succeeds for both read and
> > > write pipes if the write pipe is created by CreateNamedPipe() and the
> > > read pipe is created by CreateFile() contrary to the current create()
> > > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > > unnecessary.
> > > 
> > > Please see the revised patch attached.
> > 
> > I haven't had a chance to test this myself yet, but occurs to me that we 
> > might 
> > have a different problem after this patch: Does the write handle that we 
> > get 
> > from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
> 
> I have just checked this, and the answer is "No". Due to this problem,
> NtQueryInformationFile() call in select() fails on the write pipe.
> 
> It seems that we need more consideration...

We have two easy options:
1) Configure the pipe with PIPE_ACCESS_DUPLEX.
2) Use nt_create() again and forget C# program issue.



Even without this problem, select() for writing pipe has a bug
and does not wrok as expected. The following patch seems to be
needed.

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0..ac2fd227e 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, 
bool writing)
   that.  This means that a pipe could still block since you could
   be trying to write more to the pipe than is available in the
   buffer but that is the hazard of select().  */
-  fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
   if (fpli.WriteQuotaAvailable > 0)
{
  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,

-- 
Takashi Yano 

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown via Cygwin]

On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:

On Mon, 30 Aug 2021 09:13:14 +0900
Takashi Yano wrote:

On Sun, 29 Aug 2021 17:04:56 -0400
Ken Brown wrote:

On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:

On Sat, 28 Aug 2021 18:41:02 +0900
Takashi Yano wrote:

On Sat, 28 Aug 2021 10:43:27 +0200
Corinna Vinschen wrote:

On Aug 28 02:21, Takashi Yano via Cygwin wrote:

On Fri, 27 Aug 2021 12:00:50 -0400
Ken Brown wrote:

Two years ago I thought I needed nt_create to avoid problems when calling
set_pipe_non_blocking.  Are you saying that's not an issue?  Is
set_pipe_non_blocking unnecessary?  Is that the point of your modification to
raw_read?


Yes. Instead of making windows read function itself non-blocking,
it is possible to check if the pipe can be read before read using
PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
returned.


The problem is this:

if (PeekNamedPipe())
  ReadFile(blocking);

is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
thread from draining the pipe between the PeekNamedPipe and the ReadFile
call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
stop it via a signal.


Hmm, you are right. Mutex guard seems to be necessary like pty code
if we go this way.


I have found that set_pipe_non_blocking() succeeds for both read and
write pipes if the write pipe is created by CreateNamedPipe() and the
read pipe is created by CreateFile() contrary to the current create()
code. Therefore, not only nt_create() but also PeekNamedPipe() become
unnecessary.

Please see the revised patch attached.


I haven't had a chance to test this myself yet, but occurs to me that we might
have a different problem after this patch: Does the write handle that we get
from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?


I have just checked this, and the answer is "No". Due to this problem,
NtQueryInformationFile() call in select() fails on the write pipe.

It seems that we need more consideration...


We have two easy options:
1) Configure the pipe with PIPE_ACCESS_DUPLEX.
2) Use nt_create() again and forget C# program issue.


I vote for 2), but let's see what Corinna thinks.


Even without this problem, select() for writing pipe has a bug
and does not wrok as expected. The following patch seems to be
needed.

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0..ac2fd227e 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, 
bool writing)
that.  This means that a pipe could still block since you could
be trying to write more to the pipe than is available in the
buffer but that is the hazard of select().  */
-  fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
if (fpli.WriteQuotaAvailable > 0)
 {
   paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,



I agree.

Ken

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple