Re: Future plans for Autotools

[Richard Purdie]

Firstly, thanks for the 2.70 release! Yocto Project hasn't switched yet
as whilst we tested it and reported issues, there was a last minute
change which broke things for us. We'll get that sorted and upgrade.

I am one of the lead developers of the Yocto Project and at its heart
is OpenEmbedded which is basically a cross compiling operating system
build tool. We primarily target Linux and specialise in allowing
complete customisation of the end resulting images.

We try and solve a complex problem, pulling together all the pieces of
open source needed for an operating system including the toolchain,
cross compiling them and allowing the user to customise them in
infinitely different ways. Where the software uses autotools, we
actually regenerate the configure scripts and makefiles from the source
files before configuring and building the software. We do this since we
may need to include arch support that isn't part of the original
configure script, or to include fixes we carry to components like
libtool, which I'm say to say doesn't work properly cross compiling out
the box.

One scary statistic is that if we look at where our builds spend time,
40% of our wall clock build time is spent in "do_configure" tasks,
which is mostly running autotools and is single threaded. The compile
takes much less time as we can do things in parallel. Of that 40%,
about half of it is in autoreconf and half is running configure itself.

I did profile configure and the problem is that its basically a fork
bomb. Tons of forks to do trivial shell operations. I have less insight
how to speed up the autoreconf side of things as that mostly seems to
be in m4, although an m4 deletion party of obsolete things we don't
care about has come to mind before.

I like autotools, I've spent a lot of time debugging them but I also
know they solve some hard problems. I do wish there were more active
releases and an upstream we could engage with and get issues discussed
and resolved somehow.

I will observe that I once worked on a Makefile based build system
(buildroot) and I then somehow ultimately became the bitbake maintainer
which can be thought of as a custom python tool for parsing and dealing
with metadata. Over the years I have come to love python for handling
our recipes and configuration data. I saw suggestion of using some
other language for configure than shell and the idea sounds intriguing,
particularly after my positive experiences with python.

In terms of CI, Yocto Project had a challenge there as we have huge
possible numbers of configurations. Its taken us a few years but we do
now have a CI system I'm quite proud of. In around 6 hours we can take
a configuration and build it out from scratch for 4 principle
architectures in 32, x32/n32 and 64 bit variants for multiple init
systems, kernel versions, libc implementations and much much more. As
an idea, here is a link to one of our test reports:

https://autobuilder.yocto.io/pub/non-release/20210120-9/testresults/testresult-report.txt

That does correctly say it attempted about 2.6 million tests. Included
there are booting images under qemu for all our principle target
architectures and running the test suites included with many pieces of
software, including the testsuites with gcc, glibc, binutils, perl and
python and the ltp suite. We also cover things like reproducibility.

When we heard about the new release we did run 2.70 under this test
matrix and reported issues back so hopefully the release benefited from
this indirectly.

Doing something simpler with build testing for autoconf and other GNU
projects would be entirely possible, I mention this mainly to show what
can be done!

I do worry about the future for autotools, I see projects moving over
to alternatives and I am happy to see the performance gains we get from
it.

I'd mention that the copyright assignment situation needed for
contributions is prohibitive to people being able to participate
(including me in the past). GNU's thoughts on version control and
project interconnection is also somewhat discouraging to people in the
wider open source ecosystem although I don't follow it closely.

I've also not seen mention of it but the fact that GCC and the
toolchain use an ancient version of autoconf has always been rather sad
to me.

I do think autotools has a proud heritage but to survive into the next
decade, I do think it needs to change direction, stop supporting old
obsolete systems and embrace more new technology, much like it would
have done originally. Reducing to the lowest common denominator is ok
for a while but ultimately signs a death warrant as I've seen observed
on the libtool list by devlopers in recent years too.

Everything here is meant as constructive support of trying to find a
way to allow autotools to succeed into the future.

Cheers,

Richard

Re: Future plans for Autotools

[Richard Purdie]

On Wed, 2021-01-27 at 00:49 +, Joseph Myers wrote:
> On Tue, 26 Jan 2021, Richard Purdie wrote:
> 
> > I've also not seen mention of it but the fact that GCC and the
> > toolchain use an ancient version of autoconf has always been rather sad
> > to me.
> 
> I moved GCC from 2.64 to 2.69 in 2018 (building on Simon Marchi's 
> implementation of the same move for binutils + GDB - I expect it would 
> have been much more work without his having already done the changes to 
> the shared files and shown the sorts of changes most commonly needed 
> outside the shared files and directories).

Thanks, I hadn't realised. The only two recipes we never autoreconf are
binutils and gcc, instead we do some painful things to handle libtool
issues so we get our libtool tweaks. It sounds like we should revisit
that. I guess we were so used to not being about to do it we never
looked back at it recently.

Does that mean those projects will autoreconf more regularly if there
are autotools releases?

Cheers,

Richard

Re: Future plans for Autotools

[Richard Purdie]

On Wed, 2021-01-27 at 18:26 +, Joseph Myers wrote:
> On Wed, 27 Jan 2021, Richard Purdie wrote:
> 
> > Thanks, I hadn't realised. The only two recipes we never autoreconf are
> > binutils and gcc, instead we do some painful things to handle libtool
> > issues so we get our libtool tweaks. It sounds like we should revisit
> > that. I guess we were so used to not being about to do it we never
> > looked back at it recently.
> > 
> > Does that mean those projects will autoreconf more regularly if there
> > are autotools releases?
> 
> I'm likely to follow binutils+gdb in making autoconf/automake updates in 
> GCC.

Fair enough, I'll have to look into what our options are. I think the
libtool mismatch issues are likely why we've not looked into this
autoreconf. I know we have problems with libtool in both binutils and
gcc which mean we have heavy patches for that.

> libtool updates are trickier, and probably more relevant to GCC than to 
> binutils+gdb.  GCC is using a 2009 version of libtool (reportedly commit 
> 2c9c38d8a12eb0a2ce7fe9c3862523026c3d5622) with lots of local patches, some 
> of which may not be upstream (libtool upstream isn't very active), and 
> different interpretations of --with-sysroot mean that updating libtool in 
> GCC would also require reverting libtool commit 
> 3334f7ed5851ef1e96b052f2984c4acdbf39e20c in the new version of the libtool 
> files used in GCC (in addition to making sure that any other 
> not-yet-upstream local libtool patches are preserved).

I think this is the key part of the problem, we carry a patch to rename
the option in libtool due to the conflicts the libtool naming caused.

We do have a queue of libtool patches, I'd love to see a way of
reconciling the sysroot option issue with libtool upstream. Its a
chicken and egg problem since we're less likely to spend time on
sending the patches if we aren't likely to see them at least get into
source control. I guess I'm diverging for the autoconf list now though!

Cheers,

Richard

Re: How to "bake in" configure args

[Richard Purdie]

On Tue, 2021-03-23 at 20:16 -0500, Bob Friesenhahn wrote:
> On Tue, 23 Mar 2021, Dan Mahoney (Gushi) wrote:
> 
> > Is there a "standard" way to do this?  Google kind of fails for this.
> 
> I see that Gavin Smith already answered the question you asked, but 
> GraphicsMagick took a different tact.  The configure options used are 
> interesting but don't provide the answers for what the configure 
> script ended up using for the build.  And so GraphicsMagick is 
> providing additional data like this:
> 
> Configured using the command:
>    ../GraphicsMagick/configure  'PSDelegate=/bin/true' 'CC=gcc-10' 
> 'CXX=g++-10' 'CFLAGS=-march=native -O2 -frename-registers -g -Wall 
> -Winline -W -Wformat-security -Wpointer-arith -Wdisabled-optimization 
> -Wmissing-noreturn -Wno-unknown-pragmas -Wvla' 'CXXFLAGS=-march=native 
> -O2 -g -Wall -Winline -W -Wextra -Wno-unknown-pragmas' 
> 'LDFLAGS=-L/usr/local/lib -Wl,-rpath,/usr/local/lib' 
> '--with-quantum-depth=16' '--disable-shared' '--enable-static' 
> '--enable-openmp-slow' '--enable-silent-rules' '--without-xml'
> 
> Final Build Parameters:
>    CC   = gcc-10
>    CFLAGS   = -fopenmp -march=native -O2 -frename-registers -g -Wall 
> -Winline -W -Wformat-security -Wpointer-arith -Wdisabled-optimization 
> -Wmissing-noreturn -Wno-unknown-pragmas -Wvla -Wall -pthread
>    CPPFLAGS = -I/usr/include/freetype2
>    CXX  = g++-10
>    CXXFLAGS = -march=native -O2 -g -Wall -Winline -W -Wextra 
> -Wno-unknown-pragmas -pthread
>    LDFLAGS  = -L/usr/local/lib -Wl,-rpath,/usr/local/lib
>    LIBS = -ljbig -lwebp -lwebpmux -llcms2 -ltiff -lfreetype -ljpeg 
> -lpng16 -lwmflite -lX11 -llzma -lz -lzstd -lm -lpthread

For reference, there is a downside to doing this which is that you
can end up with non-reproducible binaries.

Yocto Project cross compiles and uses compiler commands like:

x86_64-poky-linux-gcc \
  
--sysroot=/media/build1/poky/build/tmp/work/core2-64-poky-linux/acl/2.2.53-r0/recipe-sysroot
 \
  -O2 -pipe -g -feliminate-unused-debug-types 
  
-fmacro-prefix-map=/media/build1/poky/build/tmp/work/core2-64-poky-linux/acl/2.2.53-r0=/usr/src/debug/acl/2.2.53-r0
 
  
-fdebug-prefix-map=/media/build1/poky/build/tmp/work/core2-64-poky-linux/acl/2.2.53-r0=/usr/src/debug/acl/2.2.53-r0
 
  
-fdebug-prefix-map=/media/build1/poky/build/tmp/work/core2-64-poky-linux/acl/2.2.53-r0/recipe-sysroot=
 
  
-fdebug-prefix-map=/media/build1/poky/build/tmp/work/core2-64-poky-linux/acl/2.2.53-r0/recipe-sysroot-native=
 
  -fvisibility=hidden -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z 
-Wl,relro -Wl,-z -Wl,now

which ensure that we end up with cross compiled binaries that are 
reproducible regardless of the path they're built in and that debug
objects and debug code can be correctly found. The downside is that
build paths are in CC/CFLAGS and if those are injected into any of
the output without filtering, the result depends on the build path.

The same does apply to hardcoding the configure command to a lesser
extent as there may or may not be build paths present, it depends on
the configure script in question and what inputs it needs.

In the cases where a project has added things like this, we tend to
have to patch it out.

Cheers,

Richard

Re: Parallelization of shell scripts for 'configure' etc.

[Richard Purdie]

On Tue, 2022-06-14 at 13:11 -0400, Nick Bowler wrote:
> The resulting config.h is correct but pa.sh took almost 1 minute to run
> the configure script, about ten times longer than dash takes to run the
> same script.  More than half of that time appears to be spent just
> loading the program into pa.sh, before a single shell command is
> actually executed.

Thanks for sharing that, it saves me looking into it!

I work on a cross compiling build environment (Yocto Project) and we
find that a large percentage of our build times (20%?) are in the
configure stage, either running autoreconf or configure with a 50/50
split between the two. We autoreconf since we change the macros in some
cases, e.g. libtool.

I would love to find a way to be more efficient about this part of our
builds. We do already provide some cached values for some macros to try
and be a little more efficient.

When I've profiled things, most of the time seems to be "fork" overhead
of builds having to fork new processes to run shell command pipelines.
I have sometimes wondered if we couldn't make code which was more
optimised to the common case and didn't have so much forking going on.

Cheers,

Richard

Re: On time64 and Large File Support

[Richard Purdie]

On Fri, 2022-11-11 at 08:38 +, Sam James wrote:
> In Gentoo, we've been planning out what we should do for time64 on glibc [0]
> and concluded that we need some support in glibc for a newer option. I'll 
> outline
> why below.
> 
> Proposal: glibc gains two new build-time configure options:
> * --enable-hard-time64
> * --enable-hard-lfs
> 
> These would hard-enable the relevant #defines within glibc's headers and 
> ensure that any
> binaries built with such a glibc have both Large File Support (LFS) and 
> time64 support.
> 
> I've come to the conclusion it's infeasible to try handle the migration 
> piecemeal. Various
> mismatches can and will occur (and while it's more likely with time64, it's 
> possible with LFS
> too) [1].

As a data point, Yocto Project has been debating this a bit. Based upon
what I've seen so far, I reached a similar conclusion, that we probably
needed to "just switch". As a source based system, the situation is
quite different to distros with binary feeds though.

My questions are around how much warning we can give people where code
is doing something it shouldn't and I don't have a good feel for how
much risk we're putting onto users with existing runtimes.

Cheers,

Richard

Re: new snapshot available: autoconf-2.72c

[Richard Purdie]

On Mon, 2023-03-27 at 08:38 -0700, Jim Meyering wrote:
> We're overdue for a new release, so here's a snapshot in preparation
> for that, which I want to call 2.73 (skipping 2.72).  There has never
> been an autoconf-2.72 release, yet `git describe` now prints 2.72c and
> has been printing strings like v2.72a-92-g8db00aa8 for years.
> 
> If you maintain a package that uses autoconf and can make a release soon,
> please install this and use it to bootstrap its configs.  We'd love to
> expose any issues before we make the release.  I've already done that
> with grep-3.10, so far without incident.

I work on Yocto Project which cross compiles Linux systems from
sources. We run autoreconf against most things.

I tried the 2.72c tarball and found three things which failed.

libarchive 3.6.2 at configure time:

| checking for int16_t... no
| checking for uint16_t... no
| checking for uint8_t... no
| checking for gcc  options needed to detect all undeclared functions... cannot 
detect
| configure: error: in 
'/media/build/poky/build/tmp/work/x86_64-linux/libarchive-native/3.6.2-r0/build':
| configure: error: cannot make gcc  report undeclared builtins
| See 'config.log' for more details
| NOTE: The following config.log files may provide further information.
| NOTE: 
/media/build/poky/build/tmp/work/x86_64-linux/libarchive-native/3.6.2-r0/build/config.log


gettext 0.21.1:

| autoreconf: running: aclocal 
--system-acdir=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/share/aclocal/
 
--automake-acdir=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/share/aclocal-1.16
 -I 
/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-runtime/m4
 -I 
/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-tools/m4
 --force -I m4 -I ../m4 -I gnulib-m4
| autoreconf: running: 
/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/bin/autoconf
 
--include=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-runtime/m4
 
--include=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-tools/m4
 --force
| configure.ac:113: warning: AC_RUN_IFELSE called without default to allow 
cross compiling
| ../autoconf-2.72c/lib/autoconf/general.m4:2981: AC_RUN_IFELSE is expanded 
from...
| ../autoconf-2.72c/lib/m4sugar/m4sh.m4:690: _AS_IF_ELSE is expanded from...
| ../autoconf-2.72c/lib/m4sugar/m4sh.m4:697: AS_IF is expanded from...
| ../autoconf-2.72c/lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded 
from...
| ../autoconf-2.72c/lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded 
from...
| gnulib-m4/calloc.m4:18: _AC_FUNC_CALLOC_IF is expanded from...
| gnulib-m4/calloc.m4:57: gl_FUNC_CALLOC_GNU is expanded from...
| gnulib-m4/gnulib-comp.m4:213: gl_INIT is expanded from...
| configure.ac:113: the top level
| configure.ac:113: warning: AC_RUN_IFELSE called without default to allow 
cross compiling
| ../autoconf-2.72c/lib/autoconf/general.m4:2981: AC_RUN_IFELSE is expanded 
from...
| ../autoconf-2.72c/lib/m4sugar/m4sh.m4:690: _AS_IF_ELSE is expanded from...
| ../autoconf-2.72c/lib/m4sugar/m4sh.m4:697: AS_IF is expanded from...
| ../autoconf-2.72c/lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded 
from...
| ../autoconf-2.72c/lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded 
from...
| gnulib-m4/getopt.m4:50: gl_GETOPT_CHECK_HEADERS is expanded from...
| gnulib-m4/getopt.m4:8: gl_FUNC_GETOPT_POSIX is expanded from...
| gnulib-m4/gnulib-comp.m4:213: gl_INIT is expanded from...
| configure.ac:113: the top level
| configure:8321: error: possibly undefined macro: 
_AC_SYS_LARGEFILE_TEST_INCLUDES
|   If this token and others are legitimate, please use m4_pattern_allow.
|   See the Autoconf documentation.
| autoreconf: error: 
/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/bin/autoconf
 failed with exit status: 1
| ERROR: autoreconf execution failed.


and a similar autoreconf failure from bison 3.8.2:

| autoreconf: configure.ac: not using Gtkdoc
| autoreconf: running: 
/media/build/poky/build/tmp/work/x86_64-linux/bison-native/3.8.2-r0/recipe-sysroot-native/usr/bin/autoconf
 --include=./m4 --force
| configure.ac:285: warning: AC_RUN_IFELSE called without default to allow 
cross compiling
| ../autoconf-2.72c/lib/autoconf/general.m4:2981: AC_RUN_IFELSE is expanded 
from...
| ../autoconf-2.72c/lib/m4sugar/m4sh.m4:690: _AS_IF_ELSE is expanded from...
| ../autoconf-2.72c/lib/m4sugar/m4sh.m4:697: AS_IF is expanded from...
| ../autoconf-2.72c/lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded 
from...
| ../autoconf-2.72c/lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded 
from...
| m4/calloc.m4:18: _AC_FUNC_CALLOC_IF is expanded from...
| m4/calloc.m4:57: gl_FUNC_CALLOC_G

Re: new snapshot available: autoconf-2.72c

[Richard Purdie]

On Tue, 2023-03-28 at 08:44 +0100, Richard Purdie wrote:
> On Mon, 2023-03-27 at 08:38 -0700, Jim Meyering wrote:
> > We're overdue for a new release, so here's a snapshot in preparation
> > for that, which I want to call 2.73 (skipping 2.72).  There has never
> > been an autoconf-2.72 release, yet `git describe` now prints 2.72c and
> > has been printing strings like v2.72a-92-g8db00aa8 for years.
> > 
> > If you maintain a package that uses autoconf and can make a release soon,
> > please install this and use it to bootstrap its configs.  We'd love to
> > expose any issues before we make the release.  I've already done that
> > with grep-3.10, so far without incident.
> 
> I work on Yocto Project which cross compiles Linux systems from
> sources. We run autoreconf against most things.
> 
> I tried the 2.72c tarball and found three things which failed.
> 
> libarchive 3.6.2 at configure time:
> 
> > checking for int16_t... no
> > checking for uint16_t... no
> > checking for uint8_t... no
> > checking for gcc  options needed to detect all undeclared functions... 
> > cannot detect
> > configure: error: in 
> > '/media/build/poky/build/tmp/work/x86_64-linux/libarchive-native/3.6.2-r0/build':
> > configure: error: cannot make gcc  report undeclared builtins
> > See 'config.log' for more details
> > NOTE: The following config.log files may provide further information.
> > NOTE: 
> > /media/build/poky/build/tmp/work/x86_64-linux/libarchive-native/3.6.2-r0/build/config.log
> 
> 
> gettext 0.21.1:
> 
> > autoreconf: running: aclocal 
> > --system-acdir=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/share/aclocal/
> >  
> > --automake-acdir=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/share/aclocal-1.16
> >  -I 
> > /media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-runtime/m4
> >  -I 
> > /media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-tools/m4
> >  --force -I m4 -I ../m4 -I gnulib-m4
> > autoreconf: running: 
> > /media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/bin/autoconf
> >  
> > --include=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-runtime/m4
> >  
> > --include=/media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/gettext-0.21.1/gettext-tools/m4
> >  --force
> > configure.ac:113: warning: AC_RUN_IFELSE called without default to allow 
> > cross compiling
> > ../autoconf-2.72c/lib/autoconf/general.m4:2981: AC_RUN_IFELSE is expanded 
> > from...
> > ../autoconf-2.72c/lib/m4sugar/m4sh.m4:690: _AS_IF_ELSE is expanded from...
> > ../autoconf-2.72c/lib/m4sugar/m4sh.m4:697: AS_IF is expanded from...
> > ../autoconf-2.72c/lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded 
> > from...
> > ../autoconf-2.72c/lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded 
> > from...
> > gnulib-m4/calloc.m4:18: _AC_FUNC_CALLOC_IF is expanded from...
> > gnulib-m4/calloc.m4:57: gl_FUNC_CALLOC_GNU is expanded from...
> > gnulib-m4/gnulib-comp.m4:213: gl_INIT is expanded from...
> > configure.ac:113: the top level
> > configure.ac:113: warning: AC_RUN_IFELSE called without default to allow 
> > cross compiling
> > ../autoconf-2.72c/lib/autoconf/general.m4:2981: AC_RUN_IFELSE is expanded 
> > from...
> > ../autoconf-2.72c/lib/m4sugar/m4sh.m4:690: _AS_IF_ELSE is expanded from...
> > ../autoconf-2.72c/lib/m4sugar/m4sh.m4:697: AS_IF is expanded from...
> > ../autoconf-2.72c/lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded 
> > from...
> > ../autoconf-2.72c/lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded 
> > from...
> > gnulib-m4/getopt.m4:50: gl_GETOPT_CHECK_HEADERS is expanded from...
> > gnulib-m4/getopt.m4:8: gl_FUNC_GETOPT_POSIX is expanded from...
> > gnulib-m4/gnulib-comp.m4:213: gl_INIT is expanded from...
> > configure.ac:113: the top level
> > configure:8321: error: possibly undefined macro: 
> > _AC_SYS_LARGEFILE_TEST_INCLUDES
> >   If this token and others are legitimate, please use m4_pattern_allow.
> >   See the Autoconf documentation.
> > autoreconf: error: 
> > /media/build/poky/build/tmp/work/x86_64-linux/gettext-native/0.21.1-r0/recipe-sysroot-native/usr/bin/autoconf
> >  failed with exit status: 1
> > ERROR: autoreconf execution failed.
> 
> 
> and 

Re: new snapshot available: autoconf-2.72c

[Richard Purdie]

On Tue, 2023-03-28 at 09:18 +0100, Richard Purdie wrote:
> On Tue, 2023-03-28 at 08:44 +0100, Richard Purdie wrote:
> > On Mon, 2023-03-27 at 08:38 -0700, Jim Meyering wrote:
> > > We're overdue for a new release, so here's a snapshot in preparation
> > > for that, which I want to call 2.73 (skipping 2.72).  There has never
> > > been an autoconf-2.72 release, yet `git describe` now prints 2.72c and
> > > has been printing strings like v2.72a-92-g8db00aa8 for years.
> > > 
> > > If you maintain a package that uses autoconf and can make a release soon,
> > > please install this and use it to bootstrap its configs.  We'd love to
> > > expose any issues before we make the release.  I've already done that
> > > with grep-3.10, so far without incident.
> > 
> > I work on Yocto Project which cross compiles Linux systems from
> > sources. We run autoreconf against most things.
> > 
> > I tried the 2.72c tarball and found three things which failed.
> > 
> > libarchive 3.6.2 at configure time:
> > 
> > > checking for int16_t... no
> > > checking for uint16_t... no
> > > checking for uint8_t... no
> > > checking for gcc  options needed to detect all undeclared functions... 
> > > cannot detect
> > > configure: error: in 
> > > '/media/build/poky/build/tmp/work/x86_64-linux/libarchive-native/3.6.2-r0/build':
> > > configure: error: cannot make gcc  report undeclared builtins
> > > See 'config.log' for more details
> > > NOTE: The following config.log files may provide further information.
> > > NOTE: 
> > > /media/build/poky/build/tmp/work/x86_64-linux/libarchive-native/3.6.2-r0/build/config.log

This one is caused by earlier issues in the config.log. configure.ac in
libarchive uses:

la_TYPE_UID_T

which wants to use AC_EGREP_HEADER, which in turn wants to call
$EGREP_TRADITIONAL which isn't set. That causes cascades of type issues
eventaually causing compile errors.

EGREP_TRADITIONAL does look to be set earlier in the configure script
but is probably getting unset again. It may be related to:

https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=34427e07bc9a619704816f4c333a225d9a28da15

I'm out my depth at this point.

Cheers,

Richard

Re: new snapshot available: autoconf-2.72c

[Richard Purdie]

I was able to work around the EGREP_TRADITIONAL issue by reordering
macros. The issue is conditional code blocks which mean
EGREP_TRADITIONAL was not set in some configure option combinations
leading to obtuse failures.

Our testing so far shows the following three sets of issues.


Macro ordering issues with EGREP_TRADITIONAL being unset:

apr 1.7.2
libarchive 3.6.2


Ship with an older gnulib largefile.m4 which breaks with autoconf
2.72c:

parted 3.5
man-db 2.11.2
libpipeline 1.5.7
gzip 1.12
findutils 4.9.0
bison 3.8.2
gettext 0.21.1
coreutils 9.1
wget 1.21.3

On x32 were seeing two recipes which configure "ok" but fail at compile
time:

gpgme 1.18.0
mpg123 1.31.2

https://autobuilder.yoctoproject.org/typhoon/#/builders/57/builds/6771/steps/11/logs/stdio


Now that things are roughly building, we should get some better
automated runtime testing results in a few hours.

Cheers,

Richard

Re: new snapshot available: autoconf-2.72c

[Richard Purdie]

On Tue, 2023-03-28 at 09:03 -0700, Jim Meyering wrote:
> On Tue, Mar 28, 2023 at 7:23 AM Richard Purdie
>  wrote:
> > I was able to work around the EGREP_TRADITIONAL issue by reordering
> > macros. The issue is conditional code blocks which mean
> > EGREP_TRADITIONAL was not set in some configure option combinations
> > leading to obtuse failures.
> > 
> > Our testing so far shows the following three sets of issues.
> > 
> > 
> > Macro ordering issues with EGREP_TRADITIONAL being unset:
> > 
> > apr 1.7.2
> > libarchive 3.6.2
> > 
> > 
> > Ship with an older gnulib largefile.m4 which breaks with autoconf
> > 2.72c:
> > 
> > parted 3.5
> > man-db 2.11.2
> > libpipeline 1.5.7
> > gzip 1.12
> > findutils 4.9.0
> > bison 3.8.2
> > gettext 0.21.1
> > coreutils 9.1
> > wget 1.21.3
> > 
> > On x32 were seeing two recipes which configure "ok" but fail at compile
> > time:
> > 
> > gpgme 1.18.0
> > mpg123 1.31.2
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/57/builds/6771/steps/11/logs/stdio
> > 
> > 
> > Now that things are roughly building, we should get some better
> > automated runtime testing results in a few hours.
> 
> Wow. Thank you for the thorough mass-testing and analysis.

Since we can build from source from scratch in a few hours, Yocto
Project lets us test things like this quite widely. We're hoping to do
more builds of development software in future to help try and catch
issues sooner.

>From a regression/failure point of view, the worrying issue is the
gpgme/mpg123 issue on x32 which also appears for musl 32 and 64 bit x86
targets.

https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/6881/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/6901/steps/11/logs/stdio

I've copied the final test report from our CI below which gives an idea
of the scope of coverage of our testing. The "failure"
gcc/binutils/glibc toolchain tests and the ltp failures are known
issues we need to dig deeper into over time. The basic summary is the
runtime test results were the same as usual.

Cheers,

Richard




==
Test Result Status Summary (Counts/Percentages sorted by testseries, ID)
==
--
Test Series| ID 
 | Passed   | Failed   | Skipped
--
a-full-posttrigger | oeselftest_fedora-36_qemux86_20230328201827
 | 1 (100%) | 0 (0%)   | 0 (0%) 
beaglebone | sdk_core-image-sato_i686_beaglebone-yocto_20230328153659   
 | 12 (100%)| 0 (0%)   | 0 (0%) 
buildtools | 
sdk_buildtools-extended-tarball_x86_64_qemux86-64_20230328133013
| 5 (100%) | 0 (0%)   | 0 (0%) 
buildtools | sdk_buildtools-tarball_x86_64_qemux86-64_20230328133013
 | 4 (80%)  | 0 (0%)   | 1 (20%)
edgerouter | sdk_core-image-sato_i686_edgerouter_20230328151749 
 | 12 (100%)| 0 (0%)   | 0 (0%) 
genericx86-64  | sdk_core-image-sato_i686_genericx86-64_20230328145346  
 | 12 (100%)| 0 (0%)   | 0 (0%) 
genericx86 | sdk_core-image-sato_i686_genericx86_20230328150455 
 | 12 (100%)| 0 (0%)   | 0 (0%) 
meta-arm   | sdk_core-image-sato_i686_fvp-base_20230328153357   
 | 12 (100%)| 0 (0%)   | 0 (0%) 
meta-arm   | sdk_core-image-sato_i686_generic-arm64_20230328164541  
 | 12 (100%)| 0 (0%)   | 0 (0%) 
meta-intel | sdk_core-image-sato_i686_intel-corei7-64_20230328160333
 | 12 (100%)| 0 (0%)   | 0 (0%) 
meta-mingw | 
sdk_core-image-mingw-sdktest_x86_64-mingw32_qemux86_20230328134421  
| 13 (100%)| 0 (0%)   | 0 (0%) 
multilib   | runtime_core-image-minimal_qemumips64_20230328163234   
 | 1 (1%)   | 0 (0%)   | 67 (99%)   
multilib   | runtime_core-image-sato_qemux86-64_20230328135659  
 | 36 (49%) | 0 (0%)   | 37 (51%)   
multilib   | runtime_core-image-sato_qemux86-64_20230328

Re: [platform-testers] new snapshot available: autoconf-2.72c

[Richard Purdie]

On Wed, 2023-03-29 at 01:53 +0200, Bruno Haible wrote:
> Richard Purdie wrote:
> > > We run autoreconf against most things.
> > > ...
> > > gettext 0.21.1:
> > > ...
> > 
> > The latter two look like they're as I'm missing the gnulib fixes to
> > largefile.m4.
> 
> autoreconf is not supported in GNU gettext; there's a script 'autogen.sh'
> instead.

I appreciate the official way to do it is using those scripts.

We (as in Yocto Project) are able to autoreconf most software and have
been doing this for a long time now. We have our own code to do it
generically since autogen scripts tend to do other things we don't want
such as network access which would break other project goals such as
reproducibility and license auditing. The autogen scripts tend not to
be as standardised either. About the only things we don't do this with
are gcc/binutils/gdb for historical reasons around their version
requirements for autoconf.

> When I run this script on the branch where gettext 0.21.1 has been made,
> I get this error:
> 
>   ./autogen.sh: generating configure in gettext-runtime...
>   configure:8378: error: possibly undefined macro: 
> _AC_SYS_LARGEFILE_TEST_INCLUDES
> If this token and others are legitimate, please use m4_pattern_allow.
> See the Autoconf documentation.
> 
> So, you are right that largefile.m4 from Gnulib is too old. I have now
> updated the 'largefile' and 'year2038' modules on the stable-* branches
> of Gnulib. This ensures compatibility with autoconf-2.72c.
> 
> Note: To get the fix, it's not enough to run 'autoreconf'. The upstream
> maintainer needs to pull from gnulib, then run 'bootstrap' or
> autopull.sh+autogen.sh.

Thanks, that will certainly help encourage software to have the updated
macros

Cheers,

Richard

Re: new snapshot available: autoconf-2.72c

[Richard Purdie]

On Fri, 2023-03-31 at 19:53 -0700, Paul Eggert wrote:
> On 2023-03-28 13:57, Richard Purdie wrote:
> >  From a regression/failure point of view, the worrying issue is the
> > gpgme/mpg123 issue on x32 which also appears for musl 32 and 64 bit x86
> > targets.
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/6881/steps/11/logs/stdio
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/6901/steps/11/logs/stdio
> 
> Could you explain that a bit more? All I can see from the first log is:
> 
> Summary: 2 tasks failed:
>  
> /home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/recipes-support/gpgme/gpgme_1.18.0.bb:do_compile
>  
> /home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/recipes-multimedia/mpg123/mpg123_1.31.2.bb:do_compile
> Summary: There were 4 ERROR messages, returning a non-zero exit code.
> 
> Why are these two failures worrisome and what do they indicate about 
> problems with Autoconf? I assume there are more detailed logs somewhere 
> but I don't know where. Thanks.

The full compile output is further back in the logs. Since the
configure stage passed, I had to reproduce the issue to look at the
configure stage (I could have pulled the logs).

I was worried about these two since they're the only ones occurring out
of a much larger set of software and they occur after a successful
configure, implying the configure wasn't really right.

It wasn't just musl since this also happened with a glibc x32 build.

The common factor is they're both cases where our site cache sets:

ac_cv_sys_file_offset_bits=${ac_cv_sys_file_offset_bits=64}

In these two pieces of software, it looks like this configure stops
setting -D_FILE_OFFSET_BITS=64 in these cases. The setup in gpgme and
mpg123 detects this and errors.

I'm not really sure it is a real error as such since off_t is always 64
bit on those platforms so it doesn't need the define. It does appear to
be a bit of a change of behaviour on autoconf's part. If I remove the
site cache entries, things work out ok.

Reading bug reports in the upstreams for those software, the authors
have tried very hard to be correct with large file support and are
frustrated with all the different permutations.

I suspect we're best dropping those site cache entries when we upgrade
to the new autoconf which should resolve the issue but I haven't had a
chance to 100% confirm all the details.

Cheers,

Richard

Re: autoconf-2.72d released [beta]

[Richard Purdie]

On Thu, 2023-11-30 at 16:51 -0500, Zack Weinberg wrote:
> We are pleased to announce beta release 2.72d of Autoconf.  (Versions
> 2.72a, 2.72b, and 2.72c were development snapshots, not official alpha
> or beta releases.)
> 
> 2.72 will be a minor bug-fix release.  The most significant changes
> are support for the upcoming 2024 release of the C standard, and new
> macros and configure command line options to facilitate the ongoing
> transition to 64-bit time_t.  Regrettably, support for C 2024 requires
> us to withdraw support for "traditional" (pre-1989) C compilers; if
> this is a serious problem for your use of Autoconf, please let us know
> *immediately*.  See the NEWS below for further summary of changes.
> 
> We intend to make the final release of 2.72 by the end of 2023,
> so please test this release widely.  Send feedback to
> bug-autoc...@gnu.org, or use the Savannah issue tracker
> .
> We would prefer not to make many more changes before the release.
> If you are aware of any existing bug reports that you believe
> *cannot* wait for 2.73, please contact bug-autoc...@gnu.org.

FWIW we tested this with Yocto Project and everything appears to work
fine with our core builds and there weren't any unexpected changes. We
were using 2.72c (we wanted to ensure it was tested and be ready for
the final release) and had a lot of backported patches which we could
drop. As such we've switched to the 2.72d version.

Cheers,

Richard

Re: m4p: an implementation of GNU m4 in Python

[Richard Purdie]

On Fri, 2025-05-09 at 17:45 -0400, Nikolaos Chatzikonstantinou wrote:
> I rewrote GNU m4 in Python. Long story short, I wanted to learn m4 to
> fix some issues I had with GNU Guile and Autotools, and after
> realizing m4 1.4 is ~8000 lines of code and reading e.g.
>  which claims
> "Feature gaps in GNU M4 hold back development of Autoconf." I thought
> I'd rewrite it in Rust. (It turned out to be more beneficial to
> rewrite in Python due to faster prototyping for the time being.)
> Eventually I plan to get back to my original purpose of fixing the
> integration of GNU Guile and Autotools.
> 
> You can find the project home page here,
>  and you can install with
> pip via `pip install m4p`.
> 
> I'm hoping that this is a welcome addition. I'm interested to hear if
> m4p works as a GNU m4 replacement for you, and any issues that arise.

FWIW, OpenEmbedded/Yocto Project knows how to cross compile a large
chunk of open source software and build functional images from source.
It runs autoreconf on most of the software so if you wanted to test
this against a large pool of software, it would allow that.

I'd be curious on the speed comparison if you do as autoreconf spends
most of it's time in m4.

Cheers,

Richard