Thanks for the responses - I'm yet to try some of the suggestions [and
check if they are appropriate for my use case].
On Wed, 1 Mar 2023, Paul Smith wrote:
> On Tue, 2023-02-28 at 22:34 -0600, Satish Balay via Bug reports and
> discussion for GNU make wrote:
> > And the reason we use the order '-j8 ${MAKEFLAGS}' is so that we have
> > the following behavior':
> >
> > - invoking 'make' defaults to using a default of '-j8'
> > - invoking 'make -j4' gives: '-j8 -j4' - i.e ability to override this
> > (pre-set) default [when needed].
>
> I'm pretty confident that this has never worked.
Perhaps my minimal test code is not an exact representation. However
this usage does work in our code-base.
>>>
[balay@pj01 petsc]$ make --version |head -1
GNU Make 4.3
[balay@pj01 petsc]$ grep ^MAKE_NP configure.log
MAKE_NP = 6
[balay@pj01 petsc]$ (ccache -c -C && make clean) > /dev/null && time (make
all) > /dev/null
real 1m15.591s
user 6m4.442s
sys 1m9.179s
[balay@pj01 petsc]$ (ccache -c -C && make clean) > /dev/null && time (make -j1
all) > /dev/null
real 6m5.378s
user 5m5.263s
sys 0m59.175s
[balay@pj01 petsc]$ (ccache -c -C && make clean) > /dev/null && time (make
-j12 all) > /dev/null
real 1m8.758s
user 10m58.865s
sys 1m41.094s
<<<
Note: '-j6' is the value set in the sub-makefile for all the above runs.
I guess something that might have enabled this usage is - the toplevel
makefile is set as .NOTPARALLEL
Additional note: one of the reasons for the 2 level makefiles is - the
top level can be invoked with "make" binary available in user PATH
(i.e could be bsd-make, solaris-make, or older gnumake aka MacOS
etc..) - and the sub-makefile is setup to use gnu-make (could be a
newer make on MacOS).
[and sure - with some make implementation combination - some MAKEFLAGS
don't get passed in appropriately from top-level to gnu-make in this
setup]
Satish
>
> When the initial, top-level make is invoked with -jN it creates the
> jobserver and it seeds the jobserver with N tokens to be used across
> all the sub-makes. It then passes details about the jobserver, in
> MAKEFLAGS, to all sub-makes.
>
> Any sub-make that sees the jobserver details in MAKEFLAGS will use that
> same jobserver and when they want to run a job they will retrieve a
> token from the jobserver. If they get one they run the job. Otherwise
> they wait until one is available. They pay no attention whatsoever to
> the value of N in any -jN argument that they receive and they don't
> keep track of the number of jobs they are currently running and compare
> it to this value.
>
> It is definitely not the case that you can limit (or increase) a
> specific sub-make to have a different number of jobserver tokens. The
> only way you can change this is to create a NEW jobserver, with its own
> set of tokens that are completely distinct from the original jobserver.
> And the only way to do that is to _remove_ the jobserver details from
> MAKEFLAGS; here you are explicitly _preserving_ the jobserver details
> in MAKEFLAGS.
>
> In fact, in earlier versions of GNU make we didn't even pass the -j
> option to sub-makes at all; now we do but it's really only there so
> people can query the MAKEFLAGS variable to see how many tokens were
> available. GNU Make doesn't pay any attention to this value, in a sub-
> make.
>
> I'm not saying that there isn't an issue here related to how we are now
> handling MAKEFLAGS. But, the particular use case, at least as you've
> described it here, has never worked the way you expect.
>
