Hello,
I would like to use all processor cores for a software build.
So I try to reuse a corresponding system setting by a command
like "getconf _NPROCESSORS_ONLN" in recipes for a make file.
A bit of build preparation needs to be performed in a serial
way before I can begin the desired parallel
Hello,
I would like to abort the evaluation of a make script
if data processing was requested for a target which does
not support a parallel software build process so far.
I do not want to use the .NOTPARALLEL pseudo-target for
some of my applications when I can call a make function
like "error" o
>> So I need to be flexible with the specification of input and
>> output parameters for these programs. But I see some software
>> development challenges for this use case.
>
> I'm sorry but I don't really understand what you're trying to do.
How do you think about to share any more software dev
> Jobserver is a method of communicating how many jobs make thinks are
> running between different instances (parent/child) of the make
> program itself, so it knows that no more than N jobs are invoked
> between all instances.
How do you think about to delegate such an implementation detail
to a
> I assume that you want something different from this
> but you need to explain better, sorry :-)
I hope that an other wording will be clearer.
Can make rules be extended on demand while a build script
is evaluated?
How much can a command like "getconf _NPROCESSORS_ONLN"
influence rule evaluati
> It seems like you want to automatically expand or contract the
> number of jobs that make will run in parallel,
Yes, exactly.
I can not use the parameter "-j" directly for the start of the main
make process in my case so far.
> based on some processing of the recipe.
One of the variables (or
> So you have in your toolbox $(shell) and $(eval).
I am not familiar enough with the second make function.
http://www.gnu.org/software/make/manual/html_node/Eval-Function.html
> $(eval) lets you generate rules dynamically.
Does it really support the generation of completely new build rules
(in
> There are two completely separate things: one thing is make deciding
> how many jobs can be run in parallel and when more jobs can be started.
Thanks for your explanation.
> That's controlled by the jobserver and currently there's no
> way to replace the built-in jobserver implementation with
> I guess I don't really see what the purpose of this is. If you have 2
> CPUs you get targets MYTARGET_0 and MYTARGET_1. If you have 4 CPUs you
> get targets MYTARGET_0, MYTARGET_1, MYTARGET_2, MYTARGET_3.
>
> So... then what?
I need to find some source files before I can start further process
>> The corresponding processes should be efficiently started for
>> parallel background execution after a serial data preparation
>> was finished.
>
> This makes very little sense.
This issue depends on work flow ordering.
> In a properly written Makefile, your serial preparation step
> will re
> I wrote some blog posts about eval and other metaprogramming techniques
> in make that you might find interesting:
>
> http://make.mad-scientist.net/category/metaprogramming/
I find this article also useful and helpful for my software development tasks.
You mention a construct "define … endef"
> The "define" construct is already in the index.
I am missing an explanation there for the aspect when the construct "define …
endef"
should be used instead of the other variants for variable assignments.
http://www.gnu.org/software/make/manual/html_node/Reading-Makefiles.html#index-define_002c-
> define..endef allows you to assign multi-line values to a variable.
Thanks for your clarification.
Would it help a bit to mention this information explicitly
in the manual?
http://www.gnu.org/software/make/manual/html_node/Reading-Makefiles.html#index-define_002c-expansion
Regards,
Markus
___
>> Would it help a bit to mention this information explicitly
>> in the manual?
>
> I'm not sure what you mean. If you look up "define" in the GNU make
> manual index:
>
> http://www.gnu.org/software/make/manual/make.html#Name-Index_fn_letter-D
>
> it will send you here:
>
> http://www.gnu.org
Hello,
I can generate a few file names with an approach
like the following.
MY_FILES::=
$(foreach index,$(shell seq 2 -1 0),$(eval MY_FILES+=X$(index).txt))
all: $(MY_FILES)
@echo '|$^|'
$(MY_FILES): ;
Now I am looking for a way to extract the number from
the shown prerequisite.
Can I
>> How do you think about to delegate such an implementation detail
>> to a configurable job submission system?
>
> That could be done, but the current implementation of the jobserver
> software isn't really separable like that. It would be a nice thing to
> make a more abstract API which could b
>>> So you have in your toolbox $(shell) and $(eval).
>>
>> I am not familiar enough with the second make function.
>> http://www.gnu.org/software/make/manual/html_node/Eval-Function.html
>
> I wrote some blog posts about eval and other metaprogramming techniques
> in make that you might find inte
> GNU make is the only implementation of make that supports any GNU
> make functions, including $(eval ...), if that's what you mean.
I hope that more software tools can cope with make file syntax and
processing of corresponding GNU extensions.
> In fact, recursive variable expansion is about th
> Make implementations are notorious for having lots of incompatible
> extensions. That's because the POSIX standard for make (which all
> implementations typically adhere to) is very limited in what it
> requires, so implementations have added their own features in addition
> to the standard, and
Hello,
Make files can grow to a size (with all their build rules
and various script code) where it becomes harder to follow
the involved data processing structures.
Do you know any software tools which provide graphical
visualisations for work flows that were developed with
the make file syntax?
> You can use "gmake --print-database -f Makefile" and then write a script
> to convert that into .dot format which gephi will load up for you.
Do you know a few scripts (or other software tools) which provide such
a data format conversion?
Will it become easier to present and edit an overview of
Hello,
Several parameters can be passed to user-defined make functions.
http://www.gnu.org/software/make/manual/html_node/Syntax-of-Functions.html
Some strings can become longer. So I see a need occasionally
to distribute the parameter list over more lines for better
build script readability.
I
Hello,
The rule syntax description mentions an use case
where it will be needed to work with four dollar signs.
The description of the eval function points also out
to be careful with the $ character.
Can it happen that even more "dollars" are needed
when several variables are called for evaluati
Hello,
I can use some phony targets from my build script
as expected.
But the software tool "make 4.1-2.2" does not like
a specific place in this file at the moment.
…
parallel-inc.make:247: *** missing separator. Stop.
The mentioned line has got a leading tab character
for the beginning of a
> Of course, if you had actually quoted the relevant lines from your
> Makefile, we could be more precise.
Thanks for your interest.
Do you get further ideas from an excerpt of my build script "parallel-inc.make"?
Lines 241 - 256:
$(EXIT_CODES_FUNCTIONS_THAT_CHECK_THEIR_SINGLE_PARAMETER_IN_HEADE
> So that $(eval $(call GIBBERISH)) is wrong. Exactly *how* it's wrong
> depends on what the $(call) is supposed to return. Currently you're
> evaling something that doesn't look like a *STANDALONE* chunk of
> Makefile, which results in the error *in the eval*.
I have added another tab character
>> I have added another tab character to a multi-line variable definition
>> which should be called for a specific evaluation.
>> Now I get a bit of more "progress indication" from a different error message.
>
> You're just trying random stuff and not paying attention!
I hope that I am becoming b
> Try changing eval to info
This introspection approach will not work in the moment
when I stumble on the mentioned error messages.
> and then inspect the output to see if it makes sense as a makefile.
The result should become components for recipes in my use case.
> Write small test makefile
> The rule of thumb is, if you're using $(eval ...) inside a recipe,
> you're doing something wrong.
Which make function should be used for recipe generation instead then?
> $(eval ...) is for parsing MAKEFILE content.
Do recipes belong also to each make script?
> There is only one situation
>> The result should become components for recipes in my use case.
>
> As described before, this is not right. $(eval ...) is used to
> internalize make syntax. The result of expanding $(eval ...) is the
> _empty string_, so putting it in a recipe results in NOTHING being added
> to the recipe.
> Without knowing exactly what you mean by "handling of rules with
> multiple output files" I can't say for sure, but I think it's unlikely
> that $(eval ...) can help with this.
A lot of rules can come into the situation to fiddle with multiple
output files easily.
Examples:
* Specification of s
>> * I need to split an input file into several smaller files.
>
> I don't see any way that $(eval ...) is needed or helpful for any of
> those things.
Can a make variable be set by this function within a recipe?
>> I imagine that conditional evaluation of recipes will be possible somehow,
>> w
>> Would use cases around the standard command "split" be concrete enough
>> for further clarification?
>> https://en.wikipedia.org/wiki/Split_%28Unix%29
>
> No.
I am bit surprised that you find such a tool as an unclear example
at the moment.
I am using my Python script which I developed as a c
> Will the line continuation work also with the backslash character there?
> http://www.gnu.org/software/make/manual/html_node/Splitting-Lines.html
I have found out that it works generally.
But I do not like the documented behaviour at some places of my make scripts.
"…
Outside of recipe lines,
> You need to describe your situation. Use words, but with detail.
I have adjusted my build scripts. Now I wonder about another
error message for a recipe.
parallel-inc.make:95: *** unterminated call to function 'info': missing ')'.
Stop.
Corresponding excerpt
Lines 91 - 101:
…
$(RESULT_FUNC
>> parallel-inc.make:95: *** unterminated call to function 'info': missing ')'.
>> Stop.
>
> I think the message is pretty clear: you're missing the end parenthesis
> ")" to the info function which starts at line 95:
I read my make rule in the way that the closing parentheses
is really not on t
> The error doesn't mean that the closing parentheses is missing from line 95,
I understand this information part.
> it means that the open parenthesis that it didn't match was the
> "info" function on line 95.
I find this detail hard to believe.
I hope that the current KDevelop editor does no
> It appears that KDevelop's "makefile" editing mode doesn't properly
> handle backslash continuations.
This tool might still have got few open issues around improved handling
of the make file syntax.
But I find that its visual feedback was appropriate for parentheses pairs
at least.
>> Do you r
>> I hoped somehow that another level of data indirection could be avoided.
>
> To be honest, based on what I've seen, you could do with a few more
> levels of indirection.
My knowledge around GNU make usage is evolving.
> The rules you've presented here are (to me) almost impossible to read.
>>> As I've said, it is not possible for a single variable or function
>>> reference to span multiple logical lines.
>>
>> Are there any chances to lift this restriction for make functions?
>
> It's very unlikely. The entire implementation of the parser in make at
> its most basic level is comple
> $(eval ...) is a (relatively) recently implemented, very advanced
> feature that exists only in GNU make: no other implementation of make
> has anything like it (that I'm aware of).
Since which program version are the functions "call" and "eval"
generally available?
> So you really just don't
> But a quick check of your link above shows no comments or anything
> in the makefile describing what it does,
Thanks for your look at my example.
> and I don't really understand the purpose behind all these recipes
> which consist of if-statements where the body of the recipe is made empty
> i
> GNU make has no built-in capability to use multiple machines:
How are the chances to integrate additional job submission systems?
> conceptually it may be a straightforward extension but the effort needed
> to communicate between multiple systems over a network, send and receive
> results reli
> I have nothing against doing more than that in theory, but before I'd
> agree to add something complex I'd need to understand where the existing
> method fails, and the new method would provide significant benefits.
I guess that there are some software development challenges around
the specifica
> The question is, why is it the case that these challenges, tasks, work,
> and constraints need to be handled within make itself,
> rather than farmed out to a separate process via the SHELL capability?
Does it matter to get the accounting for the processor cores right?
> Offhand I can't see an
> I tried to use the following command for parallel build but it is not working.
> *Make **–**j 7 **–**f makefile.mak build*
I find your impression strange.
> When I execute the above command, I can see that the GNU make is utilizing
> 40%-60% of CPU.
Which tools did you use to check the run ti
> Visiting folders non PARALLEL while compiling itself inside of the folders is
> IN PARALLEL.
How do you think about to traverse the affected directory hierarchy by a
dedicated
make recipe?
* Have you got any expectations about the amount of parallel work which should
be triggered
by script
Hello,
Software development projects can grow in the way that corresponding components
will be stored in several subdirectories. Such a project organisation has got
consequences on the management of some dependencies.
Now I am looking for advanced support possibilities by the make software
once m
> Relative paths are fine when including makefiles from sub-directories.
> What matters is that targets use *consistent* paths.
Thanks for your feedback.
I would like to clarify dependencies for other build system details a bit more.
> This assumes your make system is non-recursive, of course.
> Read up on the differences between builddir and srcdir, current and top_,
> relative and abs_.
> These are all preset output variables in Autoconf.
Thanks for your reminder.
I submitted a similar clarification request also for this software area.
“Management for relative paths in global buil
Hello,
I am testing the following pattern rules with the program “GNU Make 4.2.1-1.7”
on my openSUSE Tumbleweed system as I would like to adjust some areas
in affected make scripts for another evolving software tool.
…
%.cmi: %.mli
$(OCAMLC_CMD) -c $<
%.cmo: %.ml %.cmi
$(OCAMLC_C
> elfring@Sonne:~/Projekte/Coccinelle/20160205/commons> LANG=C make -d
> commands.cmo V=1
> …
> No implicit rule found for 'commands.cmo'.
> Finished prerequisites of target file 'commands.cmo'.
> Must remake target 'commands.cmo'.
> Successfully remade target file 'commands.cmo'.
> make: Nothin
> (I would guess that most of the people on the bug-make mailing list have
> never worked with ocaml and therefore don't know what needs to be done to
> build it.)
Yesterday I realised that one source source file needed a special handling
in the mentioned subdirectory.
> The snippet of Makefil
>> I have noticed a moment ago that an interface description file was missing
>> somehow for the OCaml source file in this compilation attempt.
> ...
>> I wonder then that the make tool did not give me a direct clue for a failed
>> software dependency as I am used to in other cases.
> Would you hav
Hello,
I have constructed the following small script to be executed by the program
“GNU Make 4.2.1-1.7” on my openSUSE Tumbleweed system.
define rule_pair =
name::=$(1)
$$(name:.ml=.cmo): $$(name)
$$(OCAMLC_CMD) -c $$<
$$(name:.ml=.cmx): $$(name)
$$(OCAMLOPT_CMD) -c $$<
endef
$
> Did you tell make, then, to disable all of its default rules for creating
> commands.mli or,
> for that matter, commands.cmo, by checking them out of version control?
Not directly so far. - The source file “commands.mli” is not needed
for the generation of the object module “commands.cmo” in th
Hello,
I got some special software development experiences by the usage of
the program “GNU Make 4.2.1-1.7” on my openSUSE Tumbleweed system.
The functionality “Generating prerequisites automatically” is desribed
in the manual. I would like to check again how good this works at the moment.
I am t
>> I needed another moment to become really aware that this software module
>> is a bit special. It seems that it is intended that it will be compiled
>> without a corresponding interface description file (suffix “mli”).
>
> Well, I'm totally lost. Here's why:
>
> You told make ".cmo files can b
> I believe the point of Martin's rhetorical query is "why should make treat
> the failure of *this* pattern rule any different than the failure of the
> many other pattern rules that would have permitted this compilation
> to complete?"
It seems that I need to try harder for the desired clarifi
> Using ::= in a makefile which is already dependent on GNU make is, IMO,
> pointless.
I am trying to use portable make specifications to some degree.
But it seems to be challenging to avoid the usage of all the nice
functionality which is provided as extensions also by this software
development
> Could you provide examples of the Makefiles rules you tried?
It depends on the detail level you prefer and my tendency to increase
corresponding efforts only incrementally.
> It's very difficult to provide feedback without complete information.
Would you like to add any advice to a discussion
> What Philip is saying (and I agree 100%) is that you provide the above
> example which is excellent and helpful and allows us to see exactly what
> you're doing. Thank you for that.
I was also curious in this case how the interest will evolve to clarify
the shown construction of make rules.
>
>> define my_broken_rule=
>> name::=${1}
>> ${name:.in=.txt}: ${1}
>> $${my_test_command} $$< > $$@
>> endef
>>
>> $(eval $(call my_broken_rule,MOTD.in))
>
> Yes, this will not work.
Thanks for your quick answer.
It might also be expected that this simple example just can not work
as in
>> my_test_command?=cat
>>
>> define my_rule_demo=
>> name::=${1}
>> $${name:.in=.txt}: ${1}
>> $${my_test_command} $$< > $$@
>> endef
>>
>> $(eval $(call my_rule_demo,MOTD.in))
>
> Is there something wrong with it?
The answer depends on our views about approaches how customised
make func
> Sure, it could be made clear in the documentation that either of the
> sides of the "=" could be empty.
In how many source files should corresponding information be integrated
to make these special cases better known for substitution references?
> However, there are plenty of ways to do this s
Hello,
I have tried the following small script out together with the program
“GNU Make 4.2.1-1.7” on my openSUSE Tumbleweed system.
my_compilation?=echo
my_preparation?=cat
footer?=MOTD.txt
prepared_file?=MOTD.in
MOTD%.log: MOTD%.txt MOTD%.in
${my_compilation} "$<: $$(cat ${prepared_fil
> It would have been generated if you would have called make with a
> command like:
> LANG=C make --no-builtin-rules -f ../rule-check2.make MOTD.log
I hoped that I do not need to specify another file name for such command variant
just to test a special default setting.
> and if you had a rule to
> Could you provide examples of the Makefiles rules you tried?
A rough approximation for further discussion:
i_compilation?=echo
o_compilation?=echo
a_generation?=$(o_compilation) 'Checked modules: '
parsing_c.cma: ast_c.cmo token_annot.cmo
$(a_generation) '$<' > $@
%.cmi: %.mli
>>> LANG=C make --no-builtin-rules -f parsing-rule-check1.make
>> make: *** No rule to make target 'ast_c.cmo', needed by 'parsing_c.cma'.
>> Stop.
>
> I feel like this is the same question you've already asked,
> and Philip already answered, before.
An other area was eventually easier to expla
>> It seems then that the original make scripts could treat more
>> interface descriptions as optional somehow.
>
> Without know what your original makefiles said I can't comment on that.
Does such a feedback indicate that would like to look also into the
corresponding development repository?
Ex
>> Does such a feedback indicate that would like to look also into the
>> corresponding development repository?
>
> No. At least I don't have time to review an entire development environment.
Such a choice is also fine.
> If you post the set of rules responsible for creating those files
> in t
> It would have been generated if you would have called make with a command
> like:
elfring@Sonne:~/Projekte/Bau> LANG=C make --no-builtin-rules -f
../rule-check2.make MOTD.log
make: *** No rule to make target 'MOTD.log'. Stop.
> LANG=C make --no-builtin-rules -f ../rule-check2.make MOTD.log
>> Would you like to add any more advice to this aspect?
>
> Only if you have a specific question.
I have got another software development concern.
The section “Canceling implicit rules” of the documentation mentions something
about the omission of recipes (or do they become “empty”?).
https://w
>> My pattern example does not work with the current make software in the
>> way I hoped would be occasionally convenient.
>
> No it does not. Did you read my entire previous answer?
Yes. - I replied to it twice with different information.
Regards,
Markus
___
>> Would it make sense to allow that another special syntax will denote
>> additional placeholders (similar to the handling of the percent character)?
>
> Take a look at
> https://www.gnu.org/software/make/manual/html_node/Static-Usage.html
Thanks for your link.
> It may enable what you want al
>> Would it make sense to allow that another special syntax will denote
>> additional placeholders (similar to the handling of the percent character)?
>
> Take a look at
> https://www.gnu.org/software/make/manual/html_node/Static-Usage.html
>
> It may enable what you want already.
This functiona
> This applies only to implicit rules, not explicit rules.
>
>> Can an explicit make rule also be cancelled anyhow?
>
> No.
Thanks for this clarification.
This distinction has got important consequences on how the different
make rule types can and should be ordered relative to each other.
Woul
>> * Can it eventually happen that dependencies will not be resolved if
>> target names do not contain the percent character?
>
> I can't remember, but a simple experiment should answer that !
I guess that it will be noticed once more that implicit rules will work
in an other way than static pa
>> How much can make scripts help to manage (optional) programming
>> interface descriptions?
>
> Are you asking, how can a makefile represent a prerequisite which may be
> built, but is not required to be built and it's not an error if it
> cannot be built?
How do you think about the following a
> I don't see why it makes a difference in how rule types should be ordered
A dependency graph is constructed by make scripts.
> (I assume that by "ordered" you mean which rules are defined
> first in the makefile).
Yes.
The possibility is supported to cancel implicit rules (which refer to
tar
> It's important to realize that even though the unfortunate choice of
> naming implies otherwise, static pattern rules are in all ways
> _explicit_ rules. They are not and should never be considered
> implicit rules.
Thanks for your explanation.
This terminology can occasionally trigger corresp
> I don't think that's relevant: a graph's topology isn't dependent
> on the order in which its arcs are enumerated.
I have tried to stress the construction sequence.
> When you submit a test case that demonstrates a bug,
I stumbled on two special cases depending on your view of the correspondi
> Your claim was that either explicit rules should come before implicit rules
> or vice versa.
You are more free with your choices for placement of explicit rules in make
scripts
(because you can not and will therefore not cancel them).
You should be careful then which of your explicit rules wil
> modules_without_mli::=$(filter-out $(basename $(mli_sources)),$(basename
> $(ml_sources)))
> $(info modules_without_mli)
> $(info $(sort $(modules_without_mli)))
It seems that I got this design approach working to some degree by the addition
of a few calls of make functions. I am not completely
> If you are seeing these values printed multiple times it means that the
> makefile is being parsed multiple times.
How can such additional parsing be clarified further?
Will it help to display extra data from special make variables?
> Why that might be, we can't determine from the information
> Your Makefile looks likely to rebuild .depend, a file that it includes.
> So, like Paul said:
>>> or else make has re-executed itself
>>> because this makefile or one of its
>>> included files was rebuilt;
Yes. - This can happen at the moment.
> I wonder about the information “Re-executing[1]
> If the makefile, or any of the files it includes, is updated by make
> then make will re-execute itself and re-read all the makefiles from
> scratch so it can see the updated content. When that happens,
> obviously all the $(info ...) functions etc. will be re-expanded.
Thanks for your repeated
> The OCaml makefile specifies an order in which the files should be
> compiled in the project. If you are sidestepping that ordering, then you
> are not doing things in the intended manner.
I hope that some details were clarified to some degree in the meantime.
> There is only an intent to sup
> Why do I get such an error message for a parallel software build attempt
> when the specified dependencies worked as expected for the non-parallel
> build try before?
How do you think about the corresponding update suggestion
“Improve some dependency specifications in make rules”?
https://github
> To quote the make info pages, section 10.5.1 "Introduction to Pattern Rules":
> -
> A pattern rule contains the character `%' (exactly one of them)
I interpreted this description in the way that one placeholder would be
supported
for each element in the list of prerequisites.
> …; the `%'
> MY_FILES=$(notdir $(wildcard somedir/*.ext))
I agree that this algorithm can work. But it does not fit to my imaginations
for efficient data processing.
If the desired directory on which a file filter should be applied
could be selected before, the file names would not contain references
to thi
> I can only assume you are referring to this pattern rule:
>
>> %.cmo: %.ml %.cmi
>> $(o_compilation) '$<' > $@
>
> and you're saying that if the .cmi file cannot be created, that make
> should not consider this an error.
I became also curious on how the clarification will evolve for
th
> I am also curious on how other involved contributors will react to
> a pull request for recent software developments around the discussed
> build scripts.
Would you like to add any constructive comments there?
https://github.com/coccinelle/coccinelle/pull/108
How do you think about to clarify r
Hello,
The variable “VPATH” has got a special meaning for the make software.
https://www.gnu.org/software/make/manual/html_node/General-Search.html
“…
Thus, if a file that is listed as a target or prerequisite does not exist
in the current directory, make searches …
…”
I would interpret this in
> Are you able to provide any example proving how it matters under some more
> or less rare circumstances?
I guess not at the moment.
> Or do you find the current software good enough
I dared to point some change possibilities out recently.
> and no changes needed?
It depends on your views f
Hello,
Files are stored within directories. So there is a general dependency
that the directory should exist (together with an identifier) before a file
can be changed.
Prerequisites for make rules can be specified in the way that their file names
reference a specific build directory. I have got
> At the risk of stating the obvious, note that there is a trivial
> work-around for this - there is nothing special about dot in patterns,
> so you could merge the dot into what % matches to make it non-empty:
It might be acceptable to give up a character from either end
of the prefix (or suffix)
> Here's one way to solve the problem:
> http://www.chaos.org.uk/~eddy/craft/make.html#AutoDir
Thanks for your link for information around the topic “Autogenerating (needed)
directories”.
Will it become better recognised as another design/construction pattern?
Regards,
Markus
_
Hello,
I am fiddling with a few build scripts for another free software.
elfring@Sonne:~/Projekte/Bau/OCamlbuild> rm -f src/glob_lexer.ml && LANG=C make
--no-builtin-rules V=1 src/glob_lexer.ml; LANG=C make --no-builtin-rules V=1
~/Projekte/Bau/OCamlbuild/src/glob_lexer.ml
make: *** No rule to
>> This test result seems to show that it can make a significant difference
>> if such a target was specified by a relative or absolute path.
>> (I would expect that these specifications will refer to the same file.)
>
> Keep in mind that targets are opaque strings,
I would prefer a more convenie
1 - 100 of 105 matches
Mail list logo
