Re: new feature idea: ingesting processed rulesets

[Paul Smith]

On Sun, 2023-06-11 at 12:29 +, Zoltán Turányi wrote:
> My problem is that contrary to the make wisdom of writing a single
> Makefile (to which I agree) most projects are still divided into
> parts with separate build definitions. One can debate if this is good
> or bad - for me it is a requirement to handle this case efficiently.
> (If you do not agree there is no point in discussing further.)

I wouldn't say "requirement", especially in conjunction with
"efficiently".  If people want "more efficient" they may have to be
willing to pay the up-front cost of changing their makefiles and I
think that's a defensible position.

But you are basically saying what I said in my previous email: the only
way it makes sense to try to implement something like this in GNU Make,
is if it could be done in such a way that people DON'T have to modify
their existing makefiles.

Because if they have to modify their makefiles anyway they might as
well just do the work to make them non-recursive in the first place.

> > However, I think this will be extremely difficult, because makefile
> > targets are so free-form and prerequisites and targets are not
> > actually necessary files at all.
> 
> Couldn’t these be handled by creating a "namespace" concept for
> rules? One could merge 2 Makefiles by prefixing each target with the
> directory of the Makefile. This would make dir1/all different from
> dir2/all. Also in recipes variables, pattern rules could be used from
> the Makefile describing the recipe. Would this not be possible (in
> theory, at least)?

Sure, but obviously it's not sufficient because some makefile rules DO
want to refer targets in other makefiles / directories.  I want my
target to depend on the "libfoo.a" target from some other directory. 
How does that work?

Also, how can you replace these target references if they appear inside
a recipe?  Make can't parse shell script content in recipes (if they
are even shell scripts at all; SHELL could be /usr/bin/perl or
something).  You can't just go through all the recipe text and replace
instances of the target "foo" with "dir1/foo".

We couldn't use a simple namespace naming convention like you suggest
of prefixing the directory because targets in the current makefile
could already be named "dir1/all" for example.  So you have to find a
way to talk about namespaces which don't conflict with existing naming.

Also what about things like make recipes that invoke recursive make,
_inside the same directory_.  Automake-generated makefiles do this all
the time.  If the namespace is based solely on the containing directory
is that sufficient?  Often the recursive invocation of make overrides
variables on the command line, so it seems like using the same
"namespace" wouldn't work.

And remember the premise here is that all these things have to work
WITHOUT any changes to makefiles (or Makefile.am files for automake)
themselves.

I'm not saying I know for a fact that it couldn't be done.  But it
definitely seems to me like there are lots of issues that have to be
considered and worked out, and I'm not really sure that it would really
be that much more efficient anyway.  The real efficiency gain would be
from:

(a) Not invoking multiple make processes at all... but the proposal on
the table is that we'd still invoke the make process, and it would
still parse its makefile, it would just ship the post-parsed content
back to the parent make, so that's at best equivalently efficient and
almost certainly slightly slower; and

(b) Allowing make to have a "global view" of all the rules and targets
so it can more efficiently run things in parallel... but here we're
talking about creating multiple namespaces anyway so can we really have
these multiple namespaces to separate target AND have this global view
of targets for better parallelism at the same time?

Probably what needs to happen to help answer these questions is for
someone (but, not me :)) to take a sample automake-enabled software
package, look at the makefiles generated by automake for that package,
and come up with a concrete proposal for how it might work and some
comments on how the result would be more efficient.

[bug #64339] $(filter) and $(filter-out) interpret "match" in surprising ways

[Eli Zaretskii]

URL:
  

 Summary: $(filter) and $(filter-out) interpret "match" in
surprising ways
   Group: make
   Submitter: eliz
   Submitted: Thu 22 Jun 2023 10:07:13 PM IDT
Severity: 3 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: SCM
Operating System: Any
   Fixed Release: None
   Triage Status: None


___

Follow-up Comments:


---
Date: Thu 22 Jun 2023 10:07:13 PM IDT By: Eli Zaretskii 
The GNU Make Manual says:

'$(filter-out PATTERN...,TEXT)'
 Returns all whitespace-separated words in TEXT that _do not_ match
 any of the PATTERN words, removing the words that _do_ match one or
 more.  This is the exact opposite of the 'filter' function.

 For example, given:

  objects=main1.o foo.o main2.o bar.o
  mains=main1.o main2.o

 the following generates a list which contains all the object files
 not in 'mains':

  $(filter-out $(mains),$(objects))

Note that the text doesn't really say what it means to "match" in this
context, nor what exactly is PATTERN. The example might make you believe the
"match" means "substring". That is A "matches" B if A is some substring of B.
But this is not the case. Observe:

$ make -f-
ORIG=foo123 bar-xyz baz-yes
EXCLUDE=bar foo
all:
@echo "$(filter-out $(EXCLUDE),$(ORIG))"
^D
=> foo123 bar-xyz baz-yes

That is, "foo123" does NOT match "foo", and "bar-xyz" does NOT match "bar".
The following does work, though:

$ make -f-
ORIG=./foo ./bar ./baz
EXCLUDE=%bar %foo
all:
@echo "$(filter-out $(EXCLUDE),$(ORIG))"
^D
=> ./baz

So now we perhaps understand that PATTERN is something that includes the %
wildcard somewhere. This should be explained in the manual, and the manual
should also somehow justify the special case of exact string equality, in
which case the % wildcard is not required.

IOW, having the exact string equality as the only example in this case is
misleading and confusing. I learned this the hard way, having just spent an
hour debugging my Makefile where $(filter-out) didn't do its job, because I
thought "./foo" should match "foo", based on what the manual says.








___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #64339] $(filter) and $(filter-out) interpret "match" in surprising ways

[Paul D. Smith]

Follow-up Comment #1, bug #64339 (project make):

I believe that the use of PATTERN here is intended to make you understand that
it's the same matching rules as with pattern rules or other uses of patterns,
such as the patsubst function; patterns that don't contain a "%" always
require the entire word to match identically, everywhere in GNU Make.

However it's not an excuse: definitely the docs could be more clear here.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

Re: new feature idea: ingesting processed rulesets

[David A. Wheeler]

On Sun, 2023-06-11 at 12:29 +, Zoltán Turányi wrote:
>> Couldn’t these be handled by creating a "namespace" concept for
>> rules? One could merge 2 Makefiles by prefixing each target with the
>> directory of the Makefile. This would make dir1/all different from
>> dir2/all. Also in recipes variables, pattern rules could be used from
>> the Makefile describing the recipe. Would this not be possible (in
>> theory, at least)?
> 

On Jun 22, 2023, at 2:41 PM, Paul Smith  wrote:
> Sure, but obviously it's not sufficient because some makefile rules DO
> want to refer targets in other makefiles / directories.  I want my
> target to depend on the "libfoo.a" target from some other directory. 
> How does that work?

If you want to create multiple makefiles and specify a "current namespace for 
files
in a directory's makefile", and you're using GNU make,
that's easily done by using this construct in the makefile fragment in each 
directory:
DIR := $(shell pwd)

Below is a sample of what this would look like. I'm using ".mk" as the extension
for fragments. A real makefile should generate the dependencies, not hand-jam 
them,
but hopefully you get the idea.

If you *need* portability with BSD make as well, you could use this instead:
DIR != pwd
... but if any directory includes "$" you're in trouble :-).
POSIX make is very minimal & it's usually better to just say "must use GNU 
make".

If you always run "make" from the root directory, this should work fine.
It should take less than a second for the no-work case even in larger systems
(e.g., 100K files or so being managed as dependencies).
A user *could* run "make" in just a subdirectory as long as there are
never cross-directory dependencies, using this approach.
However, it will end BADLY if there were EVER
cross-directory dependencies AND a user ran "make" using only the
makefile in a *subdirectory*. In my experience, the cross-directory
dependencies WILL eventually happen. So it's better to NEVER name them 
"makefile"
(so users won't accidentally screw up building),
and instead name these makefile fragments with a ".mk" extension.
Create your makefiles in each directory, and run "make" at the top directory,
where its "makefile" transitively includes the others.

If it takes more than than a second to run "do-nothing" at the top dir and you 
have <100K files,
then there's a serious problem that needs fixing.

If you think you *have* to run make in
a way that it can *only* see the dependencies of a subdirectory,
I think you're optimizing the wrong problem.
Make is *very* fast at parsing makefiles & at checking file timestamps.
But if you give Make the wrong data, it will do the wrong thing.
You may then try to "fix" it by overspecifying things, leading to
complex & unmaintainable makefiles that will be slow & wrong.

--- David A. Wheeler

~~~

===
./makefile
===
# Demo getting current directory for use in file references

DIR := $(shell pwd)

$(DIR)/result: $(DIR)/result.o $(DIR)/subdir/bar.o

clean:
rm -f *.o */*.o

include subdir/fragment.mk

===
./subdir/fragment.mk
===

DIR := $(shell pwd)

$(DIR)/bar.o: $(DIR)/bar.c

===
./subdir/bar.c
===
#include 

void bar(void) {
printf("In bar.\n");
}

===
./result.c
===
#include 

extern void bar(void);

int main(void) {
printf("In main.\n");
bar();
return 0;
}


> 
>> My problem is that contrary to the make wisdom of writing a single
>> Makefile (to which I agree) most projects are still divided into
>> parts with separate build definitions. One can debate if this is good
>> or bad - for me it is a requirement to handle this case efficiently.
>> (If you do not agree there is no point in discussing further.)
> 
> I wouldn't say "requirement", especially in conjunction with
> "efficiently".  If people want "more efficient" they may have to be
> willing to pay the up-front cost of changing their makefiles and I
> think that's a defensible position.
> 
> But you are basically saying what I said in my previous email: the only
> way it makes sense to try to implement something like this in GNU Make,
> is if it could be done in such a way that people DON'T have to modify
> their existing makefiles.
> 
> Because if they have to modify their makefiles anyway they might as
> well just do the work to make them non-recursive in the first place.
> 
>>> However, I think this will be extremely difficult, because makefile
>>> targets are so free-form and prerequisites and targets are not
>>> actually necessary files at all.
>> 
> 
> Also, how can you replace these target references if they appear inside
> a recipe?  Make can't parse shell script content in recipes (if they
> are even shell scripts at all; SHELL could be /usr/bin/perl or
> something).  You can't just go through all the recipe text and replace
> instances of the target "foo" with "dir1/foo".
> 
> We couldn't use a simple namespace naming convention like you suggest
> of prefixing the directory because targets in the current makef

Re: new feature idea: ingesting processed rulesets

[David Boyce]

> DIR := $(shell pwd)

My only contribution to this is to point out that I believe the above
construct, while very common, is unnecessary. The $(CURDIR) variable is
defined by the manual to be an absolute path to the current working
directory of the make process. Combined with the fact that make has no way
to change directories (recipes can cd but make itself cannot, which is a
feature), I see no way that invoking a shell and pwd can improve on simply
using $(CURDIR).

(Other) David

On Thu, Jun 22, 2023 at 5:13 PM David A. Wheeler 
wrote:

> On Sun, 2023-06-11 at 12:29 +, Zoltán Turányi wrote:
> >> Couldn’t these be handled by creating a "namespace" concept for
> >> rules? One could merge 2 Makefiles by prefixing each target with the
> >> directory of the Makefile. This would make dir1/all different from
> >> dir2/all. Also in recipes variables, pattern rules could be used from
> >> the Makefile describing the recipe. Would this not be possible (in
> >> theory, at least)?
> >
>
> On Jun 22, 2023, at 2:41 PM, Paul Smith  wrote:
> > Sure, but obviously it's not sufficient because some makefile rules DO
> > want to refer targets in other makefiles / directories.  I want my
> > target to depend on the "libfoo.a" target from some other directory.
> > How does that work?
>
> If you want to create multiple makefiles and specify a "current namespace
> for files
> in a directory's makefile", and you're using GNU make,
> that's easily done by using this construct in the makefile fragment in
> each directory:
> DIR := $(shell pwd)
>
> Below is a sample of what this would look like. I'm using ".mk" as the
> extension
> for fragments. A real makefile should generate the dependencies, not
> hand-jam them,
> but hopefully you get the idea.
>
> If you *need* portability with BSD make as well, you could use this
> instead:
> DIR != pwd
> ... but if any directory includes "$" you're in trouble :-).
> POSIX make is very minimal & it's usually better to just say "must use GNU
> make".
>
> If you always run "make" from the root directory, this should work fine.
> It should take less than a second for the no-work case even in larger
> systems
> (e.g., 100K files or so being managed as dependencies).
> A user *could* run "make" in just a subdirectory as long as there are
> never cross-directory dependencies, using this approach.
> However, it will end BADLY if there were EVER
> cross-directory dependencies AND a user ran "make" using only the
> makefile in a *subdirectory*. In my experience, the cross-directory
> dependencies WILL eventually happen. So it's better to NEVER name them
> "makefile"
> (so users won't accidentally screw up building),
> and instead name these makefile fragments with a ".mk" extension.
> Create your makefiles in each directory, and run "make" at the top
> directory,
> where its "makefile" transitively includes the others.
>
> If it takes more than than a second to run "do-nothing" at the top dir and
> you have <100K files,
> then there's a serious problem that needs fixing.
>
> If you think you *have* to run make in
> a way that it can *only* see the dependencies of a subdirectory,
> I think you're optimizing the wrong problem.
> Make is *very* fast at parsing makefiles & at checking file timestamps.
> But if you give Make the wrong data, it will do the wrong thing.
> You may then try to "fix" it by overspecifying things, leading to
> complex & unmaintainable makefiles that will be slow & wrong.
>
> --- David A. Wheeler
>
> ~~~
>
> ===
> ./makefile
> ===
> # Demo getting current directory for use in file references
>
> DIR := $(shell pwd)
>
> $(DIR)/result: $(DIR)/result.o $(DIR)/subdir/bar.o
>
> clean:
> rm -f *.o */*.o
>
> include subdir/fragment.mk
>
> ===
> ./subdir/fragment.mk
> ===
>
> DIR := $(shell pwd)
>
> $(DIR)/bar.o: $(DIR)/bar.c
>
> ===
> ./subdir/bar.c
> ===
> #include 
>
> void bar(void) {
> printf("In bar.\n");
> }
>
> ===
> ./result.c
> ===
> #include 
>
> extern void bar(void);
>
> int main(void) {
> printf("In main.\n");
> bar();
> return 0;
> }
>
>
> >
> >> My problem is that contrary to the make wisdom of writing a single
> >> Makefile (to which I agree) most projects are still divided into
> >> parts with separate build definitions. One can debate if this is good
> >> or bad - for me it is a requirement to handle this case efficiently.
> >> (If you do not agree there is no point in discussing further.)
> >
> > I wouldn't say "requirement", especially in conjunction with
> > "efficiently".  If people want "more efficient" they may have to be
> > willing to pay the up-front cost of changing their makefiles and I
> > think that's a defensible position.
> >
> > But you are basically saying what I said in my previous email: the only
> > way it makes sense to try to implement something like this in GNU Make,
> > is if it could be done in such a way that people DON'T have to modify
> > their existing makefiles.
> >
> > Because if they have to 

Re: new feature idea: ingesting processed rulesets

[David A. Wheeler]

> On Jun 22, 2023, at 5:39 PM, David Boyce  wrote:
> 
> > DIR := $(shell pwd)
> 
> My only contribution to this is to point out that I believe the above 
> construct, while very common, is unnecessary. The $(CURDIR) variable is 
> defined by the manual to be an absolute path to the current working directory 
> of the make process. Combined with the fact that make has no way to change 
> directories (recipes can cd but make itself cannot, which is a feature), I 
> see no way that invoking a shell and pwd can improve on simply using 
> $(CURDIR).

Ah, $(CURDIR) is even better, since then you don't have to remember to insert 
it in each makefile fragment.

So that sounds like the "obvious" solution unless there's something I'm missing.

--- David A. Wheeler