Re: Parallelization of shell scripts for 'configure' etc.

I've been telling folks about the config site file every time this thread comes up. Good on you for actually trying it haha. It can make a huge difference. You can short circuit a lot of checks this way. Now, the disclaimer: you still shouldn't share a cache file between projects, and if you use

Re: Parallelization of shell scripts for 'configure' etc.

Tim Rühsen writes: > a) The maintainer/contributor/hacker setup > This is when you re-run configure relatively often for the same project(s). > I do this normally and and came up with > https://gitlab.com/gnuwget/wget2/-/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain. > > It may be a

Re: Parallelization of shell scripts for 'configure' etc.

On Jun 13, 2022, Paul Eggert wrote: > In many Gnu projects, the 'configure' script is the biggest barrier to > building because it takes s long to run. Is there some way that we > could improve its performance without completely reengineering it, by > improving Bash so that it can parallelize

Re: Parallelization of shell scripts for 'configure' etc.

On 6/18/22 3:05 PM, Tim Rühsen wrote: [Configure for wget2 runs 'rm' and 'cat' each roughly 2000x - so I came up with enabling plugins for those two commands (had to write a plugin for 'rm', not sure if it never has been accepted by bash upstream).] The loadable `rm' has been in bash since ba

Re: Parallelization of shell scripts for 'configure' etc.

Hi all, On 14.06.22 00:39, Paul Eggert wrote: In many Gnu projects, the 'configure' script is the biggest barrier to building because it takes s long to run. Is there some way that we could improve its performance without completely reengineering it, by improving Bash so that it can parall

Re: Parallelization of shell scripts for 'configure' etc.

On 2022/06/13 15:39, Paul Eggert wrote: In many Gnu projects, the 'configure' script is the biggest barrier to building because it takes s long to run. Is there some way that we could improve its performance without completely reengineering it, by improving Bash so that it can parallelize '

Re: Parallelization of shell scripts for 'configure' etc.

On Thu, 16 Jun 2022, Demi Marie Obenour wrote: Finally, a small but probably noticable improvement would come from dropping support for ancient platforms, such as Ultrix. A much bigger win would be to use Bash or Zsh if they are installed, as that allows using modern shell tricks (such as [[ "$

Re: Parallelization of shell scripts for 'configure' etc.

Python and Perl carry a massive dependency closure; notably this closure depends on `autoconf` itself, so "using Python or Perl in `autoconf`" creates a large, very large, bootstrap paradox; BUT projects that aren't members of the Perl/Python closure could take advantage of those tools. We had an

Re: Parallelization of shell scripts for 'configure' etc.

On 6/14/22 16:36, Richard Purdie wrote: > On Tue, 2022-06-14 at 13:11 -0400, Nick Bowler wrote: >> The resulting config.h is correct but pa.sh took almost 1 minute to run >> the configure script, about ten times longer than dash takes to run the >> same script. More than half of that time appears

Re: Parallelization of shell scripts for 'configure' etc.

Probably my previous email went unnoticed… But as I was saying I have published a couple of m4 macros for addressing exactly this point raised

Re: Parallelization of shell scripts for 'configure' etc.

On Thu, Jun 16, 2022, at 12:13 AM, Warren Young wrote: > A Makefile can be viewed as a series of shell scripts interconnected by > dependency rules that dictate which elements can be safely skipped, which > must be run, and in what order. > > I believe it's possible to recast Autoconf so that it ge

Re: Parallelization of shell scripts for 'configure' etc.

A Makefile can be viewed as a series of shell scripts interconnected by dependency rules that dictate which elements can be safely skipped, which must be run, and in what order. I believe it's possible to recast Autoconf so that it generates a Makefile that produces the same outputs as a configure

Re: Parallelization of shell scripts for 'configure' etc.

Paul Eggert wrote: > It appears that PaSh itself isn't suitable for prime-time use. That's my current conclusion too: I tried running PaSh on - the configure script of a coreutils release, - gnulib-tool (a large shell script), and got a parse error in both cases [1]. Bruno [1] https://github

Re: Parallelization of shell scripts for 'configure' etc.

On Tue, Jun 14, 2022 at 11:15 PM Ángel wrote: > > On 2022-06-13 at 18:32 -0700, Paul Eggert wrote: > > Yes, all that could be done in theory, but it'd take a lot of > > hacking and it's been decades and it hasn't happened. > > > > I'd rather have shell scripts "just work" in parallel with a minimu

Re: Parallelization of shell scripts for 'configure' etc.

On 6/14/22 18:55, Ángel wrote: Do you have any handy example of configure that takes too long to run? Sure. Coreutils, emacs. Pretty much any nontrivial configure script takes longer than it should. I understand that parallelization of shell scripts is nontrivial.

Re: Parallelization of shell scripts for 'configure' etc.

On 2022-06-13 at 18:32 -0700, Paul Eggert wrote: > Yes, all that could be done in theory, but it'd take a lot of > hacking and it's been decades and it hasn't happened. > > I'd rather have shell scripts "just work" in parallel with a minimum > of fuss. If this hasn't happened, that might be becau

Re: Parallelization of shell scripts for 'configure' etc.

On 6/14/22 10:11, Nick Bowler wrote: The resulting config.h is correct but pa.sh took almost 1 minute to run the configure script, about ten times longer than dash takes to run the same script. More than half of that time appears to be spent just loading the program into pa.sh, before a single

Re: Parallelization of shell scripts for 'configure' etc.

On Tue, 2022-06-14 at 13:11 -0400, Nick Bowler wrote: > The resulting config.h is correct but pa.sh took almost 1 minute to run > the configure script, about ten times longer than dash takes to run the > same script. More than half of that time appears to be spent just > loading the program into p

Re: Parallelization of shell scripts for 'configure' etc.

On 14/06/2022, Nick Bowler wrote: > On 2022-06-14, Michael Orlitzky wrote: >> On Mon, 2022-06-13 at 15:39 -0700, Paul Eggert wrote: >>> >>> I've wanted something like this for *years* (I assigned a simpler >>> version to my undergraduates but of course it was too much to expect >>> them to implem

Re: Parallelization of shell scripts for 'configure' etc.

On 2022-06-14, Michael Orlitzky wrote: > On Mon, 2022-06-13 at 15:39 -0700, Paul Eggert wrote: >> >> I've wanted something like this for *years* (I assigned a simpler >> version to my undergraduates but of course it was too much to expect >> them to implement it) and I hope some sort of paralleliz

Re: Parallelization of shell scripts for 'configure' etc.

Chet Ramey wrote in <211b74c0-caed-227f-ffae-b85868ef7...@case.edu>: |On 6/13/22 6:39 PM, Paul Eggert wrote: |> In many Gnu projects, the 'configure' script is the biggest barrier to |> building because it takes s long to run. Is there some way that we |> could improve its performance with

Re: Parallelization of shell scripts for 'configure' etc.

On 6/13/22 6:39 PM, Paul Eggert wrote: > In many Gnu projects, the 'configure' script is the biggest barrier to > building because it takes s long to run. Is there some way that we > could improve its performance without completely reengineering it, by > improving Bash so that it can paralleliz

Re: Parallelization of shell scripts for 'configure' etc.

On Mon, 2022-06-13 at 15:39 -0700, Paul Eggert wrote: > > I've wanted something like this for *years* (I assigned a simpler > version to my undergraduates but of course it was too much to expect > them to implement it) and I hope some sort of parallelization like this > can get into production

Re: Parallelization of shell scripts for 'configure' etc.

Four months ago I had created a couple of m4 macros to allow parallelization of the configure process: https://github.com/madmurphy/not-autotools/blob/master/m4/not-parallel-configure.m4 If you look at the source code you will find also a sample usage: # Start a new unnamed thread NC

Re: Parallelization of shell scripts for 'configure' etc.

You can try to use the `requires` toposort routine to identify "Strongly Connected Sub-Components", which is where I imagine you'll get the best results. What you'll need to watch out for is undeclared ordering requirements that parallelism would break. The `m4sh` and `m4sugar` source code is docu

Re: Parallelization of shell scripts for 'configure' etc.

Paul Eggert writes: > In many Gnu projects, the 'configure' script is the biggest barrier to > building because it takes s long to run. Is there some way that we > could improve its performance without completely reengineering it, by > improving Bash so that it can parallelize 'configure' s

Re: Parallelization of shell scripts for 'configure' etc.

Yeah honestly splitting most of the `configure` checks into multiple threads is definitely possible. Caching between projects is even a straightforward extension with systems like `Nix`. The "gotcha" here in both cases is that existing scripts that are living in source tarballs are not feasible t

Re: Parallelization of shell scripts for 'configure' etc.

On 6/13/22 18:25, Dale R. Worley wrote: It seems to me that bash provides the needed tools -- "( ... ) &" lets you run things in parallel. Similarly, if you've got a lot of small tasks with a complex web of dependencies, you can encode that in a "makefile". It seems to me that the heavy work is

Parallelization of shell scripts for 'configure' etc.

In many Gnu projects, the 'configure' script is the biggest barrier to building because it takes s long to run. Is there some way that we could improve its performance without completely reengineering it, by improving Bash so that it can parallelize 'configure' scripts? For ideas about thi