Re: 4.4.1 breaks recursive invocation with --print-directory [when adding to MAKEFLAGS]

[Andreas Schwab]

On Feb 28 2023, Satish Balay via Bug reports and discussion for GNU  make wrote:

> On Tue, 28 Feb 2023, Dmitry Goncharov wrote:
>
>> On Tue, Feb 28, 2023 at 8:41 PM Satish Balay via Bug reports and
>> discussion for GNU make  wrote:
>> > $ cat makefile
>> > all:
>> > @MAKEFLAGS="-j1 ${MAKEFLAGS}" ${MAKE} -f makefile hello
>> > hello:
>> > @echo Hello
>> 
>> When you pass --print-directory MAKEFLAGS has the value "w".
>> When this makefile prepends -j1 the value of MAKEFLAGS becomes "-j1 w".
>> Among other changes short options are now in the first word on makeflags.
>> 
>> Well defined MAKEFLAGS has the following format
>> [shortoptions] [-option with arg]... [--long option]... [ -- cli definitions]
>> 
>> -j carries an argument and is supposed to follow short options.
>> 
>> If you change to
>> MAKEFLAGS="${MAKEFLAGS} -j1" $(MAKE) -f makefile hello
>> then it'll work.
>
> For one, the long option '--print-directory' is automatically converted to 
> this short notation - contributing to this issue.
>
> And the reason we use the order '-j8 ${MAKEFLAGS}' is so that we have the 
> following behavior':

You need to write it as MAKEFLAGS="-j1 -${MAKEFLAGS}", so that the short
options at the start of the old value of MAKEFLAGS become a proper
options string.

-- 
Andreas Schwab, SUSE Labs, sch...@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: 4.4.1 breaks recursive invocation with --print-directory [when adding to MAKEFLAGS]

[Edward Welbourne]

On Feb 28 2023, Satish Balay via Bug reports and discussion for GNU  make wrote:
>> And the reason we use the order '-j8 ${MAKEFLAGS}' is so that we have the 
>> following behavior':

Andreas Schwab (1 March 2023 10:15) wrote:
> You need to write it as MAKEFLAGS="-j1 -${MAKEFLAGS}", so that the short
> options at the start of the old value of MAKEFLAGS become a proper
> options string.

Alternatively, you can

ifeq($(filter -j%,$(MAKEFLAGS)),)
  MAKEFLAGS += -j8
endif

but that will be less portable to other make flavours,

Eddy.

Re: 4.4.1 breaks recursive invocation with --print-directory [when adding to MAKEFLAGS]

[Dmitry Goncharov]

On Tue, Feb 28, 2023 at 11:35 PM Satish Balay via Bug reports and
discussion for GNU make  wrote:
> And the reason we use the order '-j8 ${MAKEFLAGS}' is so that we have the 
> following behavior':
>
> - invoking 'make' defaults to using a default of '-j8'
> - invoking 'make -j4' gives: '-j8 -j4' - i.e ability to override this 
> (pre-set) default [when needed].
>
> Will have to see if we can reorder options in the recommended order - in our 
> intermediate makefile.


Oops, i didn't expect that you'd treat that email as "the recommended order"..
In the particular case of --print-directory and -j1, appending -j1 would work.
However, the latest make also keeps command line definitions as the
last thing in makeflags.
So, if you appended -j1 and also specified a command line definition,
e.g. v=1, then makeflags would become
-- v=1 -j1
and the submake would take -j1 as a target.
i do not recommend appending or prepending to env variable MAKEFLAGS.
In fact, i recommend against it.

Why don't you pass -j1 as the command line switch to submake?
all:; @$(MAKE) -j1 hello
hello:; @echo Hello

This would work with the old and new versions of make.

regards, Dmitry

[bug #63852] Two test failure running the test suite as root

[Paul D. Smith]

Follow-up Comment #1, bug #63852 (project make):

Well, we would have to use a separate check: whether we can write to
directories that don't have write privileges.  It could be done.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

Re: 4.4.1 breaks recursive invocation with --print-directory [when adding to MAKEFLAGS]

[Paul Smith]

On Tue, 2023-02-28 at 22:34 -0600, Satish Balay via Bug reports and
discussion for GNU make wrote:
> And the reason we use the order '-j8 ${MAKEFLAGS}' is so that we have
> the following behavior':
> 
> - invoking 'make' defaults to using a default of '-j8'
> - invoking 'make -j4' gives: '-j8 -j4' - i.e ability to override this
> (pre-set) default [when needed].

I'm pretty confident that this has never worked.

When the initial, top-level make is invoked with -jN it creates the
jobserver and it seeds the jobserver with N tokens to be used across
all the sub-makes.  It then passes details about the jobserver, in
MAKEFLAGS, to all sub-makes.

Any sub-make that sees the jobserver details in MAKEFLAGS will use that
same jobserver and when they want to run a job they will retrieve a
token from the jobserver.  If they get one they run the job.  Otherwise
they wait until one is available.  They pay no attention whatsoever to
the value of N in any -jN argument that they receive and they don't
keep track of the number of jobs they are currently running and compare
it to this value.

It is definitely not the case that you can limit (or increase) a
specific sub-make to have a different number of jobserver tokens.  The
only way you can change this is to create a NEW jobserver, with its own
set of tokens that are completely distinct from the original jobserver.
And the only way to do that is to _remove_ the jobserver details from
MAKEFLAGS; here you are explicitly _preserving_ the jobserver details
in MAKEFLAGS.

In fact, in earlier versions of GNU make we didn't even pass the -j
option to sub-makes at all; now we do but it's really only there so
people can query the MAKEFLAGS variable to see how many tokens were
available.  GNU Make doesn't pay any attention to this value, in a sub-
make.

I'm not saying that there isn't an issue here related to how we are now
handling MAKEFLAGS.  But, the particular use case, at least as you've
described it here, has never worked the way you expect.

[bug #63856] The special target .WAIT does not work as special target on command line.

[Tzvetelin Katchov]

URL:
  

 Summary: The special target .WAIT does not work as special
target on command line.
   Group: make
   Submitter: katchov
   Submitted: Wed 01 Mar 2023 03:13:15 PM UTC
Severity: 3 - Normal
  Item Group: Enhancement
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 4.4.1
Operating System: None
   Fixed Release: None
   Triage Status: None


___

Follow-up Comments:


---
Date: Wed 01 Mar 2023 03:13:15 PM UTC By: Tzvetelin Katchov 
The special target .WAIT works only as prerequisite in Makefile. When
specified on command line, it is not considered as special target.

Example:

$ make -j2 clean .WAIT all
make: *** No rule to make target '.WAIT'.  Stop.








___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #63856] .WAIT does not work as special target on command line.

[Paul D. Smith]

Update of bug #63856 (project make):

   Component Version:   4.4.1 => 4.4
 Summary: The special target .WAIT does not work as special
target on command line. => .WAIT does not work as special target on command
line.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

Re: 4.4.1 breaks recursive invocation with --print-directory [when adding to MAKEFLAGS]

[Satish Balay via Bug reports and discussion for GNU make]

Thanks for the responses - I'm yet to try some of the suggestions [and
check if they are appropriate for my use case].

On Wed, 1 Mar 2023, Paul Smith wrote:

> On Tue, 2023-02-28 at 22:34 -0600, Satish Balay via Bug reports and
> discussion for GNU make wrote:
> > And the reason we use the order '-j8 ${MAKEFLAGS}' is so that we have
> > the following behavior':
> > 
> > - invoking 'make' defaults to using a default of '-j8'
> > - invoking 'make -j4' gives: '-j8 -j4' - i.e ability to override this
> > (pre-set) default [when needed].
> 
> I'm pretty confident that this has never worked.

Perhaps my minimal test code is not an exact representation. However
this usage does work in our code-base.

>>>
[balay@pj01 petsc]$ make --version |head -1
GNU Make 4.3
[balay@pj01 petsc]$ grep ^MAKE_NP configure.log 
MAKE_NP = 6
[balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time (make  
all) > /dev/null

real1m15.591s
user6m4.442s
sys 1m9.179s
[balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time (make -j1 
all) > /dev/null

real6m5.378s
user5m5.263s
sys 0m59.175s
[balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time (make 
-j12 all) > /dev/null

real1m8.758s
user10m58.865s
sys 1m41.094s
<<<

Note: '-j6' is the value set in the sub-makefile for all the above runs.

I guess something that might have enabled this usage is - the toplevel
makefile is set as .NOTPARALLEL

Additional note: one of the reasons for the 2 level makefiles is - the
top level can be invoked with "make" binary available in user PATH
(i.e could be bsd-make, solaris-make, or older gnumake aka MacOS
etc..) - and the sub-makefile is setup to use gnu-make (could be a
newer make on MacOS).

[and sure - with some make implementation combination - some MAKEFLAGS
don't get passed in appropriately from top-level to gnu-make in this
setup]

Satish


> 
> When the initial, top-level make is invoked with -jN it creates the
> jobserver and it seeds the jobserver with N tokens to be used across
> all the sub-makes.  It then passes details about the jobserver, in
> MAKEFLAGS, to all sub-makes.
> 
> Any sub-make that sees the jobserver details in MAKEFLAGS will use that
> same jobserver and when they want to run a job they will retrieve a
> token from the jobserver.  If they get one they run the job.  Otherwise
> they wait until one is available.  They pay no attention whatsoever to
> the value of N in any -jN argument that they receive and they don't
> keep track of the number of jobs they are currently running and compare
> it to this value.
> 
> It is definitely not the case that you can limit (or increase) a
> specific sub-make to have a different number of jobserver tokens.  The
> only way you can change this is to create a NEW jobserver, with its own
> set of tokens that are completely distinct from the original jobserver.
> And the only way to do that is to _remove_ the jobserver details from
> MAKEFLAGS; here you are explicitly _preserving_ the jobserver details
> in MAKEFLAGS.
> 
> In fact, in earlier versions of GNU make we didn't even pass the -j
> option to sub-makes at all; now we do but it's really only there so
> people can query the MAKEFLAGS variable to see how many tokens were
> available.  GNU Make doesn't pay any attention to this value, in a sub-
> make.
> 
> I'm not saying that there isn't an issue here related to how we are now
> handling MAKEFLAGS.  But, the particular use case, at least as you've
> described it here, has never worked the way you expect.
> 

Re: 4.4.1 breaks recursive invocation with --print-directory [when adding to MAKEFLAGS]

[Paul Smith]

On Wed, 2023-03-01 at 13:05 -0600, Satish Balay wrote:
> Perhaps my minimal test code is not an exact representation. However
> this usage does work in our code-base.
> 
> > > > 
> [balay@pj01 petsc]$ make --version |head -1
> GNU Make 4.3
> [balay@pj01 petsc]$ grep ^MAKE_NP configure.log 
> MAKE_NP = 6
> [balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time
> (make  all) > /dev/null
> 
> real1m15.591s
> user6m4.442s
> sys 1m9.179s
> [balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time
> (make -j1 all) > /dev/null
> 
> real6m5.378s
> user5m5.263s
> sys 0m59.175s
> [balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time
> (make -j12 all) > /dev/null
> 
> real1m8.758s
> user10m58.865s
> sys 1m41.094s
> <<<
> 
> Note: '-j6' is the value set in the sub-makefile for all the above
> runs.

All of this behavior makes sense and aligns with what I wrote in my
previous email message, with one caveat I didn't think of.

What happens is this:

In the first instance where you don't provide -j at all, the sub-make
(that adds -j6) will create a new jobserver with 6 tokens and that make
and all its sub-makes will share that jobserver.

In the second instance where you provide -j1, the arguments passed to
the sub-make (after you've modified them) will be: "-j6 -j1" because "-
j1" appears in makeflags.  I didn't consider this idea of explicitly
passing "-j1" on the command line.

In the third instance where you provide -j12, a new jobserver will be
created with 12 tokens and the top-level make and all its sub-makes
will share  that jobserver, regardless of the value of -j added by the
sub-make.

So, the only time that the MAKEFLAGS="-j6 $$MAKEFLAGS" has a different
behavior than MAKEFLAGS="$$MAKEFLAGS -j6" is in the case where someone
invokes "make -j1" at the top level; in the first case this will
prevent the sub-make from starting a jobserver, while in the second
case it will still do that.  Otherwise it has no effect.

You can prove this to yourself with this simple makefile:

$ cat Makefile
.RECIPEPREFIX := >
top:
> @echo "$@: $$MAKEFLAGS"
> $(MAKE) middle

middle:
> @echo "$@: $$MAKEFLAGS"
> MAKEFLAGS="-j4 $$MAKEFLAGS" $(MAKE) bottom

bottom: 1 2 3 4
> @echo "$@: $$MAKEFLAGS"

1 2 3 4:
> @echo $@ start; sleep 2; echo $@ end


First we run make with no -j at all (this is GNU make 4.3 FYI):

$ make --no-print-directory
top:  --no-print-directory
make middle
middle:  --no-print-directory
MAKEFLAGS="-j4 $MAKEFLAGS" make bottom
1 start
2 start
3 start
4 start
1 end
2 end
3 end
4 end
bottom:  -j4 --jobserver-auth=3,4 --no-print-directory

You can see from the output that no jobserver was available in the
"middle" invocation, and that the "bottom" invocation was run with a
jobserver with 4 slots available (all four jobs started in parallel).

Now if we run with -j1:

$ make --no-print-directory -j1
top:  -j1 --no-print-directory
make middle
middle:  -j1 --no-print-directory
MAKEFLAGS="-j4 $MAKEFLAGS" make bottom
1 start
1 end
2 start
2 end
3 start
3 end
4 start
4 end
bottom:  -j1 --no-print-directory

Now, the invocation of the jobserver in "middle" was ignored because
the -j1 option on the command line overrode the -j4 in the "middle"
rule.  And we can see the "bottom" targets were all run serially, not
in parallel.

Finally, add -j2 to the parent:

$ make --no-print-directory -j2
top:  -j2 --jobserver-auth=3,4 --no-print-directory
make middle
middle:  -j2 --jobserver-auth=3,4 --no-print-directory
MAKEFLAGS="-j4 $MAKEFLAGS" make bottom
1 start
2 start
1 end
2 end
3 start
4 start
3 end
4 end
bottom:  -j2 --jobserver-auth=3,4 --no-print-directory


Here you can see that the "-j4" option in "middle" was ignored.  But if
you reverse the MAKEFLAGS assignment:

> MAKEFLAGS="$$MAKEFLAGS -j4" $(MAKE) -f /tmp/x8.mk bottom

You'll get the same results:

1 start
2 start
1 end
3 start
2 end
4 start
3 end
4 end

It still only runs with -j2 even though -j4 "should" take precedence.