Hello,
GNU make allows the user to control the number of simultaneous running
jobs using by the -j, --jobs option, and to prevent new jobs from
starting based on the load average by using the -l, --load-average
option. These options currently provide very limited means for users to
prevent the system from running out resources, most notably available
memory[1].
Adding new --some-resource options in addition to --load-average does
not seem to be very reasonable, as it might lead to in endless
discussions about which resources to account for and how exactly to do
that. For flexibility, I believe such decisions are best left to the user.
Would it please be possible to enhance GNU make with a command-line
option taking a string argument to specify a name or path to an external
program to further control starting of jobs or recipes?
Here are some of my raw ideas on this.
One could for example add an option similar to -l which would take the
name or path of an executable to executed whenever the load average
would be checked with -l. Whether or not the respective job should be
started could depend on the exit status of that program. Any auxiliary
information to the program could be provided via command-line arguments
or (prefixed) environment variables.
It were possible to extend the -l option to parse a non-number argument
as a name or path to such a program, but the long option name
"--load-average" would in my opinion not best suit such use. According
to the make(1) manpage the limit set using -l is only applied when
"there are others jobs running" (should read "other"?), meaning it can
not be used to delay execution of all jobs as in case of only one job it
will not be subject to this limit.
It would perhaps also be possible extend the -j option to take an
executable (e.g. `-j nproc`) so that new jobs would not be started if we
are already running more jobs than a number returned by the external
program. Note that it is also possible for the external program to not
exit immediately, but to wait on resources to become available. I must
say I haven't yet figured out a set of good semantics for this one.
Another way might be to provide a launcher executable (e.g. via
--launcher or similar) to be used in place of the shell to execute the
Makefile recipes, so that the executable could fully control each such
recipe invocation. Whenever make would execute a recipe, it would
instead pass the recipe (or parts thereof) and other relevant
information (including SHELL and .SHELLFLAGS) to the launcher, which
will then run the recipe. If the launcher detects insufficient resources
it can delay actual execution of the recipe until sufficient resources
become available. The launcher could then be used in addition to any -j
and -l options (with or without arguments). The launcher could also be
used for other things besides job control, such as debugging or
measuring resource usage. This idea is partly borrowed from CMake which
has RULE_LAUNCH_(COMPILE|LINK|CUSTOM) properties to allow, for example,
for easy use of ccache.
Not being too familiar with the intricacies of GNU make, my take on this
might be naive, but I'm sure other users would also appreciate such a
feature. Thanks! :)
Best regards,
Jaak Ristioja
[1]: https://stackoverflow.com/questions/11639524/make-j-ram-limits
