Re: Regression on Cygwin: Problems with parallel make in 4.4

[Ken Brown]

On 2/21/2023 12:27 AM, Paul Smith wrote:

On Mon, 2023-02-20 at 20:21 -0500, Ken Brown wrote:

Parallel make is still not working reliably.  I've just discovered
that my TeX Live build logs have several occurrences of the following
warning:

    jobserver unavailable: using -j1.  Add '+' to parent make rule.

This has been going on since I first started using 4.4.0.90 with the
jobserver style set to "pipe".  It *only* occurs when the jobserver
uses a pipe.  Two others in the Cygwin community are reporting the
same thing when they build projects using cmake (with Unix Makefiles
and make 4.4.0.91).


Just to note, I do run the regression test suite with the equivalent of
"jobserver-style=pipe" (basically I force the configure to believe that
the system doesn't support mkfifo() so it falls back to "pipe" mode).

Can you run builds with GNU Make 4.4 and --jobserver-style=pipe?  Do
you see the same warnings there?  I just want to know if the problem
happened in 4.4 or post-4.4.  If the latter, that's very odd.  If the
former, it's more understandable because we did make the "pipe" mode
more strict in 4.4.


I do see it in 4.4 with --jobserver-style=pipe.  I misspoke when I said 
I first saw it in 4.4.0.90.



This message means only one thing: the parent make didn't provide the
proper environment (that is, the correct file descriptors) to its sub-
make.  However, there could be different ways in which the environment
is wrong.

If you have the ability, it would be helpful if you could apply the
attached patch to your build of GNU Make and try the build again, and
let me know which of these different messages are generated right
before the above error.


I'll be able to do that later today.


It would also be very helpful if someone could provide me with the
complete rule that the parent make uses to invoke the sub-make, where
that message appears.  And, if you can include the output that make
generates to invoke the submake (the command it prints) that would be
excellent as well.


I'll try, again later.

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Paul Smith]

On Tue, 2023-02-21 at 08:35 -0500, Ken Brown wrote:
> On 2/21/2023 3:54 AM, Ken Brown wrote:
> > On 2/21/2023 12:27 AM, Paul Smith wrote:
> > > Just to note, I do run the regression test suite with the
> > > equivalent of "jobserver-style=pipe" (basically I force the
> > > configure to believe that the system doesn't support mkfifo() so
> > > it falls back to "pipe" mode).
> 
> Somehow the test suite isn't catching this problem, at least on
> Cygwin.

Understood.  I'm just saying it's not the case that there is no testing
at all with "pipe", anymore.  Agreed that the regression tests are not
completely sufficient to find all problems.

> make[5]: not recursive -2,-2
> make[5]: warning: jobserver unavailable: using -j1.  Add '+' to
> parent make rule.

Perfect, thanks.  That probably means the message is the result of an
explicit change we made to lock down the pipe mode, rather than some
unanticipated modification.  I'll look at the makefile you provided to
see if I can figure it out.  Maybe the changes we made need to be
adjusted.

For more info, see this entry in the NEWS file:

* When the pipe-based jobserver is enabled and GNU Make decides it is invoking
  a non-make sub-process and closes the jobserver pipes, it will now add a new
  option to the MAKEFLAGS environment variable that disables the jobserver.
  This prevents sub-processes that invoke make from accidentally using other
  open file descriptors as jobserver pipes.  For more information see
  https://savannah.gnu.org/bugs/?57242 and https://savannah.gnu.org/bugs/?62397

The "option that disables the jobserver" is where the -2,-2 comes from.

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Paul Smith]

On Tue, 2023-02-21 at 08:35 -0500, Ken Brown wrote:
> make  check-TESTS
> make[4]: Entering directory 
> '/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea'
> make[5]: Entering directory 
> '/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea'

OK, I see the rule that generates the [5] recursion:

  $(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__set_TESTS_bases); \
...

(I think).  Hm, this rule does contain $(MAKE) so it should be
automatically marked for recursion.  Maybe something weird about
AM_MAKEFLAGS?

Sorry for the iteration but can you run the build again, this time with
the "--trace" option and also editing the above rule (at line 1755 of
kpathsea/Makefile to add "set -x" as in:

  $(TEST_SUITE_LOG): $(TEST_LOGS)
@$(set -x; am__set_TESTS_bases); \
...

There will likely be a lot of output.

Thanks!

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Ken Brown]

On 2/21/2023 9:39 AM, Paul Smith wrote:

On Tue, 2023-02-21 at 08:35 -0500, Ken Brown wrote:

make  check-TESTS
make[4]: Entering directory
'/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea'
make[5]: Entering directory
'/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea'


OK, I see the rule that generates the [5] recursion:

   $(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__set_TESTS_bases); \
 ...

(I think).  Hm, this rule does contain $(MAKE) so it should be
automatically marked for recursion.  Maybe something weird about
AM_MAKEFLAGS?

Sorry for the iteration but can you run the build again, this time with
the "--trace" option and also editing the above rule (at line 1755 of
kpathsea/Makefile to add "set -x" as in:

   $(TEST_SUITE_LOG): $(TEST_LOGS)
@$(set -x; am__set_TESTS_bases); \
 ...

There will likely be a lot of output.


Here it is.

KenMakefile:570: update target 'check-recursive' due to: target is .PHONY
fail=; \
if (target_option=k; case ${target_option-} in ?) ;; *) echo 
"am__make_running_with_option: internal error: invalid" "target option 
'${target_option-}' specified" >&2; exit 1;; esac; has_opt=no; 
sane_makeflags=$MAKEFLAGS; if { if test -z '0'; then false; elif test -n 
'x86_64-pc-cygwin'; then true; elif test -n '4.4.0.91' && test -n 
'/home/kbrown/src/texlive/test.x86_64/Work'; then true; else false; fi; }; then 
sane_makeflags=$MFLAGS; else case $MAKEFLAGS in *\\[\ \   ]*) bs=\\; 
sane_makeflags=`printf '%s\n' "$MAKEFLAGS" | sed "s/$bs$bs[$bs $bs   ]*//g"`;; 
esac; fi; skip_next=no; strip_trailopt () { flg=`printf '%s\n' "$flg" | sed 
"s/$1.*$//"`; }; for flg in $sane_makeflags; do test $skip_next = yes && { 
skip_next=no; continue; }; case $flg in *=*|--*) continue;; -*I) strip_trailopt 
'I'; skip_next=yes;; -*I?*) strip_trailopt 'I';; -*O) strip_trailopt 'O'; 
skip_next=yes;; -*O?*) strip_trailopt 'O';; -*l) strip_trailopt 'l'; 
skip_next=yes;; -*l?*) strip_trailopt 'l';; -[dEDm]) skip_next=yes;; -[JT]) 
skip_next=yes;; esac; case $flg in *$target_option*) has_opt=yes; break;; esac; 
done; test $has_opt = yes); then \
  failcom='fail=yes'; \
else \
  failcom='exit 1'; \
fi; \
dot_seen=no; \
target=`echo check-recursive | sed s/-recursive//`; \
case "check-recursive" in \
  distclean-* | maintainer-clean-*) list='auxdir/auxsub doc texk/kpathsea 
texk/ptexenc  libs utils texk' ;; \
  *) list='. doc texk/kpathsea texk/ptexenc  libs utils texk' ;; \
esac; \
for subdir in $list; do \
  echo "Making $target in $subdir"; \
  if test "$subdir" = "."; then \
dot_seen=yes; \
local_target="$target-am"; \
  else \
local_target="$target"; \
  fi; \
  (CDPATH="${ZSH_VERSION+.}:" && cd $subdir && make  $local_target) \
  || eval $failcom; \
done; \
if test "$dot_seen" = "no"; then \
  make  "$target-am" || exit 1; \
fi; test -z "$fail"
Making check in .
make[1]: Entering directory '/home/kbrown/src/texlive/test.x86_64/Work'
Makefile:1002: update target 'recurse' due to: target is .PHONY
fail= failcom='exit 1'; \
for f in x $MAKEFLAGS; do \
  case $f in \
*=* | --[!k]*);; \
*k*) failcom='fail=yes';; \
  esac; \
done; \
list='texk/kpathsea texk/ptexenc '; for one_dir in $list; do \
  if test ! -f $one_dir/Makefile; then \
test -d $one_dir || /usr/bin/mkdir -p $one_dir; \
cmd=`cat subsubdir-conf.cmd | sed "s,auxdir/auxsub,$one_dir,g"`; \
case " texk/kpathsea texk/ptexenc  " in \
  *" $one_dir "*) skip=;; \
  *) skip=' --disable-build';; \
esac; \
(cd $one_dir && echo "=== configuring in $one_dir (`pwd`)" && \
  if :; then echo "make: running /bin/sh $cmd$skip"; fi && \
  CONFIG_SHELL=/bin/sh && export CONFIG_SHELL && \
  eval /bin/sh $cmd$skip  || \
{ echo "=== configuring in $one_dir failed"; exit 1; }) && \
if test "x$skip" = x; then \
  echo "Making all in $one_dir" && \
  (cd $one_dir && make  all); \
fi || eval $failcom; \
  fi; \
done; test -z "$fail"
make[1]: Leaving directory '/home/kbrown/src/texlive/test.x86_64/Work'
Making check in doc
make[1]: Entering directory '/home/kbrown/src/texlive/test.x86_64/Work/doc'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory '/home/kbrown/src/texlive/test.x86_64/Work/doc'
Making check in texk/kpathsea
make[1]: Entering directory 
'/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea'
Makefile:2096: update target 'check' due to: target is .PHONY
make  check-recursive
make[2]: Entering directory 
'/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea'
Makefile:1642: update target 'check-recursive' due to: target is .PHONY
fail=; \
if (target_option=k; case ${target_option-} in ?) ;; *) echo 
"am__make_running_with_option: internal error: invalid" "target option 
'${target_option-}' specified" >&2; exit 1;; esac; has_opt=no; 
sane_makeflags=$MAKEFLAGS; if { if test -z '2'; then false; elif test -n 
'x86_64-pc-cygwin'; then true; elif test -n '4.4.0.91' && test -n 
'/home/kbrown/src/texlive/test.x86_64/

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Paul Smith]

On Tue, 2023-02-21 at 10:12 -0500, Ken Brown wrote:
> > There will likely be a lot of output.
> 
> Here it is.

Oops I got the rule wrong for the set -x.

But, I think I might have found the bug.  If I'm right, it's a doozy! 
But, as you mentioned it's not widespread because it only affects pipe
jobservers.  I'm glad 4.4.1 is almost ready to go.  I'll try to write a
test case for it to see if I'm right.

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Paul Smith]

On Tue, 2023-02-21 at 10:36 -0500, Paul Smith wrote:
> But, I think I might have found the bug.  If I'm right, it's a doozy!
> But, as you mentioned it's not widespread because it only affects
> pipe jobservers.  I'm glad 4.4.1 is almost ready to go.  I'll try to
> write a test case for it to see if I'm right.

I was able to write a regression test that shows the bug.

Can you please apply the change below and see if it fixes the problem?
I don't think this is the final solution I'll apply for 4.4.1 but it
will let me know if I'm on the right track.


diff --git a/src/job.c b/src/job.c
index 94df0197..d054ae1b 100644
--- a/src/job.c
+++ b/src/job.c
@@ -1422,7 +1422,8 @@ start_job_command (struct child *child)
 #ifndef _AMIGA
   /* Set up the environment for the child.  */
   if (child->environment == 0)
-child->environment = target_environment (child->file, child->recursive);
+child->environment = target_environment (child->file,
+ child->file->cmds->any_recurse);
 #endif

 #if !defined(__MSDOS__) && !defined(_AMIGA) && !defined(WINDOWS32)

[bug #63821] Switch -r fails to disable default suffix rules.

[Dmitry Goncharov]

Follow-up Comment #7, bug #63821 (project make):

> We can reconsider it.

There is no need. It is all good.

I see that you removed parameter void from install_default_suffix_rules
definiton. Going forward, do you want   empty paranthesis rather than void in
functions definition?


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #63821] Switch -r fails to disable default suffix rules.

[Paul D. Smith]

Follow-up Comment #8, bug #63821 (project make):

Yes I'm going to go through and remove them all as a cleanup, not until after
this release though.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Ken Brown]

On 2/21/2023 11:34 AM, Paul Smith wrote:

On Tue, 2023-02-21 at 10:36 -0500, Paul Smith wrote:

But, I think I might have found the bug.  If I'm right, it's a doozy!
But, as you mentioned it's not widespread because it only affects
pipe jobservers.  I'm glad 4.4.1 is almost ready to go.  I'll try to
write a test case for it to see if I'm right.


I was able to write a regression test that shows the bug.

Can you please apply the change below and see if it fixes the problem?
I don't think this is the final solution I'll apply for 4.4.1 but it
will let me know if I'm on the right track.


I think you're on the right track.  I got through 'make -j13 check' 
without the jobserver warning.


Ken

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Paul Smith]

On Tue, 2023-02-21 at 13:11 -0500, Ken Brown wrote:
> I think you're on the right track.  I got through 'make -j13 check' 
> without the jobserver warning.

w00t!  Thanks for the help.  A full fix should be in 4.4.1 which I hope
to release this week or weekend.

Re: Regression on Cygwin: Problems with parallel make in 4.4

[Ken Brown]

On 2/21/2023 1:26 PM, Paul Smith wrote:

On Tue, 2023-02-21 at 13:11 -0500, Ken Brown wrote:

I think you're on the right track.  I got through 'make -j13 check'
without the jobserver warning.


w00t!  Thanks for the help.  A full fix should be in 4.4.1 which I hope
to release this week or weekend.


Just as a little icing on the cake, I tried my cmake build of doxygen, 
and again it went through with no jobserver warnings.


Ken