[bug #64746] Exponential Runtime in make 4.4.1 when export is used
URL: <https://savannah.gnu.org/bugs/?64746> Summary: Exponential Runtime in make 4.4.1 when export is used Group: make Submitter: tback Submitted: Wed 04 Oct 2023 09:31:42 PM UTC Severity: 3 - Normal Item Group: Bug Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: 4.4.1 Operating System: POSIX-Based Fixed Release: None Triage Status: None ___ Follow-up Comments: --- Date: Wed 04 Oct 2023 09:31:42 PM UTC By: Till Backhaus This is my first report here, I might lack the proper etiquette. I'm on fedora 38 which installs gnu-make 4.4.1. I'm reporting a regression related to the changed behavior of export: > * WARNING: Backward-incompatibility! > Previously makefile variables marked as export were not exported to commands > started by the $(shell ...) function. Now, all exported variables are > exported to $(shell ...). If this leads to recursion during expansion, then > for backward-compatibility the value from the original environment is used. > To detect this change search for 'shell-export' in the .FEATURES variable. This rather short Makefile shows exponential runtime depending on the number of variables. export VAR_1 ?= $(shell echo 1) VAR_2 ?= $(shell echo 2) VAR_3 ?= $(shell echo 3) VAR_4 ?= $(shell echo 4) VAR_5 ?= $(shell echo 5) VAR_6 ?= $(shell echo 6) VAR_7 ?= $(shell echo 7) foo: echo foo Runtime with 7 variables is ~20s on reasonably current hardware (AMD Ryzen 7 4750U): time make foo echo foo foo real 0m21.663s user 0m4.082s sys 0m18.027s make foo --debug=verbose GNU Make 4.4.1 Built for x86_64-redhat-linux-gnu Copyright (C) 1988-2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Reading makefiles... Reading makefile 'Makefile'... Updating makefiles Updating goal targets Considering target file 'foo'. File 'foo' does not exist. Finished prerequisites of target file 'foo'. Must remake target 'foo'. echo foo Makefile:7: not recursively expanding VAR_5 to export to shell function ... Makefile:4: not recursively expanding VAR_2 to export to shell function foo Successfully remade target file 'foo'. The number of expansions grows exponentially: I modified the Makefile to contain different numbers of variables and got an exponential number of suppressed expansions: make foo --debug=verbose | grep -c recurs 82201 Variables to Expansions: 1 var 1 2 vars 6 3 vars 33 4 vars 196 5 vars 1305 6 vars 9786 7 vars 82201 Best regards, Till ___ Reply to this item at: <https://savannah.gnu.org/bugs/?64746> ___ Message sent via Savannah https://savannah.gnu.org/
Exponential Runtime in make 4.4.1 when export is used
Hello, This is my first report here, I might lack the proper etiquette. I'm on fedora 38 which installs gnu-make 4.4.1. I'm reporting a regression related to the changed behavior of export: * WARNING: Backward-incompatibility! Previously makefile variables marked as export were not exported to commands started by the $(shell ...) function. Now, all exported variables are exported to $(shell ...). If this leads to recursion during expansion, then for backward-compatibility the value from the original environment is used. To detect this change search for 'shell-export' in the .FEATURES variable. This rather short Makefile shows exponential runtime depending on the number of variables. export VAR_1 ?= $(shell echo 1) VAR_2 ?= $(shell echo 2) VAR_3 ?= $(shell echo 3) VAR_4 ?= $(shell echo 4) VAR_5 ?= $(shell echo 5) VAR_6 ?= $(shell echo 6) VAR_7 ?= $(shell echo 7) foo: echo foo Runtime with 7 variables is ~20s on reasonably current hardware (AMD Ryzen 7 4750U): time make foo echo foo foo real 0m21.663s user 0m4.082s sys 0m18.027s make foo --debug=verbose GNU Make 4.4.1 Built for x86_64-redhat-linux-gnu Copyright (C) 1988-2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Reading makefiles... Reading makefile 'Makefile'... Updating makefiles Updating goal targets Considering target file 'foo'. File 'foo' does not exist. Finished prerequisites of target file 'foo'. Must remake target 'foo'. echo foo Makefile:7: not recursively expanding VAR_5 to export to shell function ... Makefile:4: not recursively expanding VAR_2 to export to shell function foo Successfully remade target file 'foo'. The number of expansions grows exponentially: I modified the Makefile to contain different numbers of variables and got an exponential number of suppressed expansions: make foo --debug=verbose | grep -c recurs 82201 Variables to Expansions: 1 var 1 2 vars 6 3 vars 33 4 vars 196 5 vars 1305 6 vars 9786 7 vars 82201 Best regards, Till
[bug #64746] Exponential Runtime in make 4.4.1 when export is used
Follow-up Comment #2, bug #64746 (project make):
That explanation was really helpful. I discovered the difference between
recursively expanded variables and simple expanded variables only after
posting this issue, but I was still confused why recursive expanded variables
would cause this behavior.
[comment #1 comment #1:]
> Prefer simply expanded variables with $(shell).
>
> Given that sush questions keep arising, maybe we should update NEWS with a
warning about recursively expanded variables in combination with $(shell).
Yes, I think that's a great idea.
I'd also like to add a little more context about my use case: I use this to
read an .env file and then define variables that are still undefined.
-include .env
export
BUILD_OS ?= $(shell uname -s)
...
Up until 4.4 this wasn't a problem because $(shell) didn't see exported
variables. What I'd like to have in 4.4 would be "conditional simple
expansion" from a new ?:= operator that works like this:
ifeq ($(origin VAR_1), undefined)
VAR_1 := $(shell echo 1)
endif
Until then I can use the verbose form from above, or I could introduce
another variable to do it in two lines:
VAR_1_ := $(shell echo 1)
VAR_1 ?= ${VAR_1_}
___
Reply to this item at:
___
Message sent via Savannah
https://savannah.gnu.org/
Re: Exponential Runtime in make 4.4.1 when export is used
Thank you Eddy, Prefix: I reported the issue via email and the issue tracker (didn't find the signup button at first, sent an email, found the signup button) where it's filed as https://savannah.gnu.org/bugs/index.php?64746. What Dimitry and you Eddy suggested definitely makes makes sense. Yet, I wish there was a "conditional simple expansion" operator. (https://savannah.gnu.org/bugs/index.php?64746#comment2). https://github.com/photoprism/photoprism/blob/eb44d637f3594f328131fdd0e3e987f5ca41e8bd/Makefile is the Makefile in question. (I'm not affiliated with photoprism, I'm just a user.) https://github.com/photoprism/photoprism/discussions/3790 is the discussion I had with the project. Best regards, Till Am Mo., 9. Okt. 2023 um 17:09 Uhr schrieb Edward Welbourne < edward.welbou...@qt.io>: > On Wed, Oct 4, 2023 at 6:09 PM Till Backhaus wrote: > >> This rather short Makefile shows exponential runtime depending on the > number of variables. > >> VAR_1 ?= $(shell echo 1) > > Dmitry Goncharov (6 October 2023 17:35) replied, ending: > > Prefer simply expanded variables with $(shell). > > and if you really need the "only set if not already set" aspects of ?=, > use a suitable if check around the immediate assignments, e.g. > > ifeq($(VAR_1),) > VAR_1 := $(shell echo 1) > endif > > (and, generally, avoid using $(shell ...) for anything that can be > achieved using other make built-in functions, but I assume your real > Makefile has something more complicated as the shell command). > > Eddy. >
