Testing gnulib on riscv64 - replacement modules (update)

[Richard W.M. Jones]

Back in March I was trying to test gnulib on RISC-V using qemu:

https://lists.gnu.org/archive/html/bug-gnulib/2018-03/threads.html#00022

We now have real hardware so I am trying to do the tests again,

I still get stuck on the "replacement modules" step in the
instructions.

  "Verify that Gnulib has built no replacement/workaround code
  (gllib/*.o files) - if so, this indicates problems in the libc."

and yes we do indeed have these files, many of them:

$ ls gllib/*.o
gllib/allocator.o gllib/fpurge.ogllib/printf-frexpl.o
gllib/areadlinkat.o   gllib/freading.o  gllib/printf-frexp.o
gllib/areadlink.o gllib/fseek.o gllib/printf-parse.o
gllib/asnprintf.o gllib/fseeko.ogllib/pt_chown.o
gllib/basename-lgpl.o gllib/fseterr.o   gllib/renameat2.o
gllib/careadlinkat.o  gllib/full-write.ogllib/safe-write.o
gllib/c-ctype.o   gllib/getprogname.o   gllib/save-cwd.o
gllib/chdir-long.ogllib/gettime.o   gllib/sig-handler.o
gllib/chmodat.o   gllib/hard-locale.o   gllib/sockets.o
gllib/chownat.o   gllib/integer_length.ogllib/statat.o
gllib/cloexec.o   gllib/ioctl.o gllib/stat-time.o
gllib/c-strcasecmp.o  gllib/localcharset.o  gllib/strerror_r.o
gllib/c-strncasecmp.o gllib/localename.ogllib/stripslash.o
gllib/dirname-lgpl.o  gllib/localtime-buffer.o  gllib/strnlen1.o
gllib/dup-safer-flag.ogllib/malloca.o   gllib/sys_socket.o
gllib/dup-safer.o gllib/math.o  gllib/tempname.o
gllib/exitfail.o  gllib/mbrlen.ogllib/timespec.o
gllib/fclose.ogllib/mbrtowc.o   gllib/tmpdir.o
gllib/fcntl.o gllib/nanosleep.o gllib/unistd.o
gllib/fd-hook.o   gllib/nonblocking.o   gllib/utimens.o
gllib/fd-safer-flag.o gllib/openat-die.ogllib/vasnprintf.o
gllib/fd-safer.o  gllib/openat-proc.o   gllib/wctype-h.o
gllib/fflush.ogllib/pipe-safer.ogllib/xsize.o
gllib/filenamecat-lgpl.o  gllib/printf-args.o

$ (cd gllib; ls -1 $(${GNULIB_CHECKOUT}/posix-modules | sed -e 's|-posix$||' | 
sort -u | grep -v 'nonblocking' | sed -e 's|$|.o|') 2>/dev/null )
fclose.o
fcntl.o
fflush.o
fseek.o
fseeko.o
ioctl.o
math.o
mbrlen.o
mbrtowc.o
nanosleep.o
strerror_r.o
sys_socket.o
unistd.o
wctype-h.o

Full build log:

  http://oirase.annexia.org/tmp/build.log

We're using glibc 2.27.9000-7.fc28.riscv64 from Fedora, and the latest
gnulib from git.

It looks like the glob.o bug has been fixed but there is still a lot
of investigation to be done.  BTW the real hardware has working
floating point.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: Testing gnulib on riscv64 - replacement modules (update)

[Bruno Haible]

Hi Rich,

> $ (cd gllib; ls -1 $(${GNULIB_CHECKOUT}/posix-modules | sed -e 's|-posix$||' 
> | sort -u | grep -v 'nonblocking' | sed -e 's|$|.o|') 2>/dev/null )
> fclose.o
> fcntl.o
> fflush.o
> fseek.o
> fseeko.o
> ioctl.o
> math.o
> mbrlen.o
> mbrtowc.o
> nanosleep.o
> strerror_r.o
> sys_socket.o
> unistd.o
> wctype-h.o
> 
> Full build log:
> 
>   http://oirase.annexia.org/tmp/build.log
> 
> We're using glibc 2.27.9000-7.fc28.riscv64 from Fedora, and the latest
> gnulib from git.
> 
> It looks like the glob.o bug has been fixed

Yes. And for the rest of the .o files, the explanations from my previous
analysis [1] hold.

> but there is still a lot of investigation to be done.

No, the other issues are known Linux or glibc problems that are not likely
to be fixed soon.

> BTW the real hardware has working floating point.

Yes, otherwise some floating-point modules would be appearing here or failing
their tests.

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnulib/2018-03/msg00057.html

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Bruno Haible]

Paul Smith wrote in
:
> first, GNU make provides a
> bootstrapping script that will let you compile make on systems which
> don't already have make... that means that I need to be able to build
> all these extra files without the assistance of automake (I do run
> configure).  I'm not sure how easy that will be.

To cope for this case, I would suggest to generate a "build all at once"
script from the generated Makefiles and config.status.
The user would then
  - run "./configure --disable-dependency-tracking"
  - execute the "build all at once" script generator,
  - execute the resulting script.

The output of "make -n" on the GNU make directory is quite simple.
In the case of make-4.2.1, it is:
$ make -n
[lots of automake junk]
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o ar.o 
ar.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
arscan.o arscan.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
commands.o commands.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
default.o default.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o dir.o 
dir.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
expand.o expand.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
file.o file.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
function.o function.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
getopt.o getopt.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
getopt1.o getopt1.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
guile.o guile.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
implicit.o implicit.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o job.o 
job.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
load.o load.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
loadapi.o loadapi.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
main.o main.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
misc.o misc.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
posixos.o posixos.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
output.o output.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
read.o read.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
remake.o remake.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
rule.o rule.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
signame.o signame.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
strcache.o strcache.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/usr/local/lib\" 
-DINCLUDEDIR=\"/usr/local/include\" -DHAVE_CONFIG_H -I.  -g -O2 -c -o 
variable.o variable.c
gcc -DLOCALEDIR=\"/usr/local/share/locale\" -DLIBDIR=\"/u

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Bruno Haible]

Paul Smith wrote:
> Here's what I find extremely difficult to manage regarding gnulib's
> current design:
> 
>  - Heavy reliance on .in files that need to be transformed.
> 
> On systems that aren't POSIX and don't have ready access to sed or
> similar tools, this is very annoying to deal with.
> 
> The original design of autoconf, where there was a single config.h
> which defined a ton of #define values that were then tested by other
> files, was far simpler for non-traditional systems: you merely had to
> create a pre-defined config.h file for such a target (most of the time
> these targets were homogeneous enough across releases that they didn't
> require autoconf anyway) and arrange to have the file renamed to start
> the build process.
> 
> However with gnulib's current design for the replacement system headers
> (at least) instead of using the old-style '#if !HAVE_BOOL' (as on
> example from stdbool.in.h), they use:
> 
>   #  if !@HAVE__BOOL@

I think you are barking up the wrong tree.

In the other mail I've explained how you can reduce the number of build
systems you use from 3 (Autoconf+Automake, Windows nmake, VMS nms) to
one (Autoconf+Automake).

The step that you underestimate here is "you merely had to
create a pre-defined config.h file for such a target". Gnulib-generated
header files (such as glob.h) use many more autoconf tests (such as
whether a function exists, is declared, has specific bugs). If someone
tries to create a config.h file by hand for an exotic platform, not
only will it be time-consuming (the gettext-tools config.h.in for example
has more than 400 results of configure findings), but also mistakes will
show up as compilation errors. (This is the downside of the many features
gnulib has:
  - C++ support,
  - support for many platforms,
  - using the function name 'rpl_foo' if and only if 'foo' would collide
with system-provided 'foo'.
The downside is that wrong guesses for the HAVE_* symbols lead to
compilation failures more quickly.)

Really, the approach of maintaining a config.h for a particular platform
by hand is outdated.

Even if gnulib would accommodate your wish to have a special copy of
glob.in.h that uses HAVE_FOO instead of @HAVE_FOO@, that would not solve
the problem how to maintain the config.h for specific platforms.

> On systems that aren't POSIX and don't have ready access to sed or
> similar tools

On such systems the essential step is to create the set of tools that
are needed to be able to run configure scripts. This includes a POSIX-like
shell, the coreutils, grep, sed, and maybe awk.

I would argue that porters who are "active" but have not yet achieved this
step should do this first, instead of you (and GNU) spend time on supporting
them better by adding a second or third build system to your package.

I mean, nowadays, we are running DOS inside a browser, compiling C to
JavaScript, emulating m68k on x86, and so on. The task of making the
POSIX build tools work on a non-POSIX system is much smaller and very
achievable, especially given enough time (like for VMS).

And when new OSes appear, with a roughly ANSI C compliant libc, they
typically
  - either have the POSIX build tools already ported,
  - or are supported as a cross-compilation target.

Bruno

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Bruno Haible]

PS: I wrote
> The task of making the
> POSIX build tools work on a non-POSIX system is much smaller and very
> achievable, especially given enough time (like for VMS).

The VMS people apparently actually achieved this task; the binaries of
these tools are for download at https://sourceforge.net/projects/gnv/ .

So, the same approach as for porting to native Windows should also
work for VMS.

Bruno

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Paul Smith]

On Wed, 2018-05-16 at 11:29 +0200, Bruno Haible wrote:
> This approach with "build_w32.bat" is outdated. The modern approach is
> to use the Autoconf and Automake generated configure and Makefile.in
> files without modifications.

> More in detail: About ca. 10 years ago, Automake started to include
> wrapper scripts (shell scripts) that make the MSVC compiler and archiver
> ('cl' and 'lib') be usable with the usual command-line options of a
> 'cc' and 'ar' program. From this point on, compilation for MSVC could
> be done with just a Cygwin or MSYS build environment (that includes
> bash, coreutils, grep, sed, and make).

Unfortunately (perhaps only for me) this isn't sufficient.

GNU make is a foundational tool: you need it before you can build any
other package.  It is also widely used by itself in environments where
no other POSIX-like or GNU tools are used (so, no MSYS etc.)

One could argue that since an exe for GNU make already exists for
Windows it's OK to rely on it to extract yourself from a catch-22
situation, but I'm not willing to relax that requirement.

My position continues to be that the build of GNU make on Windows (or
other environments) needs to be possible with only native tools: on
Windows this means command.com and a C compiler.  No Cygwin, no MSYS,
no bash, no sed, no grep, and obviously no make (since that's what
we're trying to build).

> On such systems the essential step is to create the set of tools that
> are needed to be able to run configure scripts. This includes a
> POSIX-like shell, the coreutils, grep, sed, and maybe awk.

And how does one do this, without make itself?

>   - No more need to care about two different build systems
> (automake on one side, nmake on the other side).

GNU make doesn't (or won't, in the next release) support nmake builds,
or any other variation of make.  They weren't used, so they have likely
been broken for a long time anyway.

> You really want to reduce the number of redundant Makefiles you have
> to maintain to 1, as soon as possible.

The only makefiles the GNU make distribution will support will be
standard make makefiles.  The model is:

If you have a configure-capable system, use autotools to compile GNU
make.

If you don't have a configure-capable system, use the provided
bootstrap script (or create your own) to build GNU make.

The build of GNU make is straightforward enough that using a script
isn't a big problem.

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Paul Eggert]

On 05/16/2018 05:43 AM, Paul Smith wrote:

The only makefiles the GNU make distribution will support will be
standard make makefiles.  The model is:

If you have a configure-capable system, use autotools to compile GNU
make.

If you don't have a configure-capable system, use the provided
bootstrap script (or create your own) to build GNU make.


Such a bootstrap script should be able to use the approach I mentioned 
earlier, by copying FOO.h.in files to FOO.h and replacing "@" with 
"_GLat" or something similar, and then by augmenting config.h with the 
appropriate #defines. If you can't assume 'sed', you can compile and run 
a simple program that does the "@" replacement.


Alternatively, we could change Gnulib to use "_GLat" instead of "@" 
uniformly, so that you don't need to do the "@" replacement in GNU Make; 
but before doing that I'd like to know that the approach actually works 
for GNU Make.

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Eli Zaretskii]

> From: Bruno Haible 
> Cc: Eli Zaretskii , Paul Eggert 
> Date: Wed, 16 May 2018 12:03:44 +0200
> 
> I think you are barking up the wrong tree.

I'm not sure.  Make is not like any other GNU package, because it is
one of the basic tools required to build other tools.  IOW, building
Make could be part of bootstrapping a GNU environment on a platform
where it was never done before.

That is why GNU Make includes shell scripts such as build.sh, even for
Posix hosts.  The ports of Make to non-Posix hosts simply don't
support anything except the build,sh-like script, that's all.

So let me turn the table and ask you how can GNU Make be built on a
Posix-ish system that lacks (or might lack) glob and/or fnmatch, if
those are imported from Gnulib?

> The step that you underestimate here is "you merely had to
> create a pre-defined config.h file for such a target". Gnulib-generated
> header files (such as glob.h) use many more autoconf tests (such as
> whether a function exists, is declared, has specific bugs). If someone
> tries to create a config.h file by hand for an exotic platform, not
> only will it be time-consuming (the gettext-tools config.h.in for example
> has more than 400 results of configure findings), but also mistakes will
> show up as compilation errors. (This is the downside of the many features
> gnulib has:
>   - C++ support,
>   - support for many platforms,
>   - using the function name 'rpl_foo' if and only if 'foo' would collide
> with system-provided 'foo'.
> The downside is that wrong guesses for the HAVE_* symbols lead to
> compilation failures more quickly.)

All this is true, but Paul is concerned only by a few specific
platforms, and about a single programming language.  So the above
features are mostly irrelevant in the case in point.

> Really, the approach of maintaining a config.h for a particular platform
> by hand is outdated.

Nevertheless, GNU Make uses it to this day.  It works for Make because
the package is in stable state, and the amount of commits that
introduce new features is very small.  Given the age of Make and the
amount of packages out there that depend on its features, it is
extremely unlikely that this situation will ever change significantly.
So the amount of effort needed to maintain a config.h is very small,
almost negligible.

> > On systems that aren't POSIX and don't have ready access to sed or
> > similar tools
> 
> On such systems the essential step is to create the set of tools that
> are needed to be able to run configure scripts. This includes a POSIX-like
> shell, the coreutils, grep, sed, and maybe awk.

That's not enough.  One also needs to tweak the configure script to
DTRT on such platforms.  As someone who did this very job for Emacs, I
can tell you that it could be a non-trivial job, although it probably
will be simpler for Make than it was for Emacs.

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Bruno Haible]

Hi Paul,

> > You really want to reduce the number of redundant Makefiles you have
> > to maintain to 1, as soon as possible.
> 
> The only makefiles the GNU make distribution will support will be
> standard make makefiles.  The model is:
> 
> If you have a configure-capable system, use autotools to compile GNU
> make.
> 
> If you don't have a configure-capable system, use the provided
> bootstrap script (or create your own) to build GNU make.
> 
> The build of GNU make is straightforward enough that using a script
> isn't a big problem.

Using a script to compile the various .c files files in turn is easy.

The problem is to maintain the config.h file, if you don't want to
assume a configure-capable system.

> My position continues to be that the build of GNU make on Windows (or
> other environments) needs to be possible with only native tools: on
> Windows this means command.com and a C compiler.  No Cygwin, no MSYS,
> no bash, no sed, no grep, and obviously no make (since that's what
> we're trying to build).

What are the reasons for this position?

- You want users of Windows ME (from 2000) to be able to build 'make'?

- You want the process of building 'make' from source to be as simple
  as possible, the least effort for the person who does it?

- You care about reproducible builds and bootstrapping, as in [1]?

- Other?

Bruno

[1] https://reproducible-builds.org/events/athens2015/bootstrapping/

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Eli Zaretskii]

> From: Paul Smith 
> Cc: bug-gnulib@gnu.org
> Date: Tue, 15 May 2018 18:31:05 -0400
> 
> Here's what I find extremely difficult to manage regarding gnulib's
> current design:
> 
>  - Heavy reliance on .in files that need to be transformed.
> 
> On systems that aren't POSIX and don't have ready access to sed or
> similar tools, this is very annoying to deal with.
> 
> The original design of autoconf, where there was a single config.h
> which defined a ton of #define values that were then tested by other
> files, was far simpler for non-traditional systems: you merely had to
> create a pre-defined config.h file for such a target (most of the time
> these targets were homogeneous enough across releases that they didn't
> require autoconf anyway) and arrange to have the file renamed to start
> the build process.
> 
> However with gnulib's current design for the replacement system headers
> (at least) instead of using the old-style '#if !HAVE_BOOL' (as on
> example from stdbool.in.h), they use:
> 
>   #  if !@HAVE__BOOL@
> 
> This means I can't just define HAVE_BOOL in config.h and then use this
> header file, I need to come up with some way of emulating sed behavior
> on all my supported platforms (Windows, VMS, etc.) so I can replace
> these values.  Or else I have to create per-system instances of each of
> these files, of which I already have 5 just for alloca and getloadavg
> and if I do take on glob/fnmatch that number will balloon.

For the record: what are those 5 systems?  MS-Windows is one, but what
are the others?

> For glob.in.h in gnulib it's even worse; even the header guards require
> replacement!!
> 
>#ifndef _@GUARD_PREFIX@_GLOB_H
> 
> I understand that the goal is to have versions of these standard header
> files which can be used without config.h, but the GNU coding standards
> suggest that config.h must be included first in each compilation unit,
> before even system headers, so to me that desire doesn't outweigh the
> downside of using this method for non-traditional systems.
> 
> I'm interested in anyone's opinion on how best to deal with this issue.

Here's an (100% untested) idea:

  . We ignore all the Gnulib headers that work around problems in
standard headers, such as stdbool.h, stdint.h, etc.  We assume the
corresponding system headers are "good enough".  Where that
happens to be false, we will add the necessary stuff to the
config.h templates.

  . For the other Gnulib headers and for config.h, we run the
configure script once, after importing those Gnulib headers, on
systems where the tools for that are available, be that a
cross-compiling environment or a native one.  (I can volunteer to
do that for MS-Windows' MinGW port.)  This only matters for Gnulib
headers that are processed at configure time; AFAICT, only glob.h
and fnmatch.h need that, since all the other Gnulib headers are
provided ready to be used, i.e. not as *.in.h templates.

  . We then keep the produced config.h and other generated Gnulib
headers in the Make repository, for each affected system, in the
same way we did with config.h templates until now.

  . The C sources of glob.c and fnmatch.c we import from Gnulib and
try not to make local changes, so that these sources will not need
any manual maintenance outside of the Gnulib project.

This idea assumes that glob.in.h and fnmatch.in.h will not change
significantly with time.  Whenever they do change, we will need to
update the templates; hopefully the changes will be minor and we will
be able to do that manually, without resorting to the full configure
run.

WDYT?

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Paul Smith]

On Wed, 2018-05-16 at 19:33 +0300, Eli Zaretskii wrote:
> > these values.  Or else I have to create per-system instances of
> > each of these files, of which I already have 5 just for alloca and
> > getloadavg and if I do take on glob/fnmatch that number will
> > balloon.
> 
> For the record: what are those 5 systems?  MS-Windows is one, but
> what are the others?

Sorry for not being clear: I meant 5 replacement header files that
contain @-tokens, not 5 supported target systems.

The systems we ostensibly support are POSIX/UNIX, Windows, VMS, MS-DOS
(?), and AmigaOS (?).  I don't know if anyone actually uses either of
the two latter platforms anymore, but the last time I asked about
ditching Amiga support (for example) there were people who objected. 
That was some years ago now though.  I've never been entirely clear on
the difference between the DOS and Windows ports.

Regarding your suggestion, that might work.  I'll look into it.

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Bruno Haible]

Eli Zaretskii wrote:
> > This is the downside of the many features gnulib has:
> >   - C++ support,
> >   - support for many platforms,
> >   - using the function name 'rpl_foo' if and only if 'foo' would collide
> > with system-provided 'foo'.
> > The downside is that wrong guesses for the HAVE_* symbols lead to
> > compilation failures more quickly.)
> 
> All this is true, but Paul is concerned only by a few specific
> platforms, and about a single programming language.  So the above
> features are mostly irrelevant in the case in point.

But we don't have two gnulibs: one with many features, and a separate one
with as few HAVE_* symbols as possible.

> > Really, the approach of maintaining a config.h for a particular platform
> > by hand is outdated.
> 
> Nevertheless, GNU Make uses it to this day.

However, gnulib modules can add new HAVE_* symbols and configuration tests
at any moment, and without notice in the NEWS file.

If Paul is OK to update (or let his porters update) the config.h files
for native Windows, VMS, and so on, each time he upgrades to a newer
version of gnulib, then fine. If not, he either shouldn't use gnulib,
or he should change the way he works with config.h.

Bruno

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Bruno Haible]

Eli Zaretskii wrote:
> Here's an (100% untested) idea:
> ...
> This idea assumes that glob.in.h and fnmatch.in.h will not change
> significantly with time.  Whenever they do change, we will need to
> update the templates; hopefully the changes will be minor and we will
> be able to do that manually

This sounds like a good compromise between Paul's requirements and the
wish to have little manual work.

When you implement it, take a look at the gnulib-tool options:
  --avoid to avoid a module despite a dependency,
  --copy-file to copy a single file if you don't want to slurp the
  entire module it belongs to.

When you upgrade to a newer version of gnulib, you should look not only
at the changes that were made in glob.in.h but also those in glob.m4.

Bruno

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Eli Zaretskii]

> From: Paul Smith 
> Cc: egg...@cs.ucla.edu, bug-gnulib@gnu.org
> Date: Wed, 16 May 2018 12:53:22 -0400
> 
> On Wed, 2018-05-16 at 19:33 +0300, Eli Zaretskii wrote:
> > > these values.  Or else I have to create per-system instances of
> > > each of these files, of which I already have 5 just for alloca and
> > > getloadavg and if I do take on glob/fnmatch that number will
> > > balloon.
> > 
> > For the record: what are those 5 systems?  MS-Windows is one, but
> > what are the others?
> 
> Sorry for not being clear: I meant 5 replacement header files that
> contain @-tokens, not 5 supported target systems.

Ah, thanks for clarifications.

> The systems we ostensibly support are POSIX/UNIX, Windows, VMS, MS-DOS
> (?), and AmigaOS (?).

MS-DOS has glob and fnmatch in its library, so it probably doesn't
need those from Gnulib.

> I don't know if anyone actually uses either of the two latter
> platforms anymore, but the last time I asked about ditching Amiga
> support (for example) there were people who objected.  That was some
> years ago now though.

"MS-DOS" is a misnomer in this case: it really means the DJGPP tools
(www.delorie.com/djgpp).  And yes, that is still used, albeit by a
very small group of people.  In particular, I use the DJGPP port of
GNU Make (and other DJGPP ports of GNU software) to build the "MS-DOS"
port of Emacs.

> I've never been entirely clear on the difference between the DOS and
> Windows ports.

They target 2 different operating systems.  Because Windows versions
up to XP included either DOS itself or its good emulation, executables
that needed DOS services could run on Windows, and I believe that is
the source of your confusion.  But since modern versions of Windows
are 64-bit, they no longer support running MS-DOS executables, and the
quality of MS-DOS subsystems on 32-bit versions of modern Windows
systems consistently deteriorates, so soon one will need a true DOS
system or a true emulator to run those.

Re: Gnulib on Windows (native / mingw32) / VMS / etc.

[Bruno Haible]

Paul Eggert wrote:
> If you can't assume 'sed', you can compile and run 
> a simple program that does the "@" replacement.

Yes, given that the uses of 'sed' in gnulib are nearly all of the form
  sed -e 's/@VAR1@/VALUE1/g' ... -e 's/@VARn@/VALUEn/g' FILE
you could write a program 'atsubst' such that
  atsubst VAR1 VALUE1 ... VARn VALUEn FILE
is equivalent to
  sed -e 's/@VAR1@/VALUE1/g' ... -e 's/@VARn@/VALUEn/g' FILE

This 'atsubst' program is simple enough that it fits in a single file
(like 'envsubst' in GNU gettext); it can be a prerequisite single command
for anyone who builds 'make'.

> Alternatively, we could change Gnulib to use "_GLat" instead of "@" 
> uniformly, so that you don't need to do the "@" replacement in GNU Make;

I don't think this would fully work. Regarding @INCLUDE_NEXT@ for example:
You can't have a C macro that expands to either #include or #include_next.

So, Paul would probably have to generate foo.halfsubsted.h from foo.in.h
where @GUARD_PREFIX@, @INCLUDE_NEXT@, @PRAGMA_SYSTEM_HEADER@, @PRAGMA_COLUMNS@
and @NEXT_FOO_H@ have been already substituted, and use C preprocessor
expansion (i.e. HAVE_FOO instead of @HAVE_FOO@) for the other symbols.

I would see this machinery as being in GNU make only (no modifications in
gnulib), unless/until the GNU bash, coreutils, grep, gawk, sed projects want
to use it as well.

Bruno

[PATCH] crypto/af_alg: fix --help

[Paul Eggert]

* m4/af_alg.m4: Avoid spurious newline in --help output.
---
 ChangeLog| 5 +
 m4/af_alg.m4 | 6 +++---
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 24fd4da80..dd479f783 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2018-05-16  Paul Eggert  
+
+   crypto/af_alg: fix --help
+   * m4/af_alg.m4: Avoid spurious newline in --help output.
+
 2018-05-13  Bruno Haible  
 
nl_langinfo: Fix compilation error on Android.
diff --git a/m4/af_alg.m4 b/m4/af_alg.m4
index db7150e08..325ab1319 100644
--- a/m4/af_alg.m4
+++ b/m4/af_alg.m4
@@ -1,4 +1,4 @@
-# af_alg.m4 serial 3
+# af_alg.m4 serial 4
 dnl Copyright 2018 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -33,8 +33,8 @@ AC_DEFUN_ONCE([gl_AF_ALG],
   use_af_alg=yes
   AC_ARG_WITH([linux-crypto],
 [AS_HELP_STRING([[--without-linux-crypto]],
-   [Do not use Linux kernel cryptographic API (default is to use it if 
available)])
-],
+   [Do not use Linux kernel cryptographic API
+(default is to use it if available)])],
 [use_af_alg=$withval],
 [use_af_alg=yes])
   dnl We cannot use it if it is not available.
-- 
2.17.0

lib/Makefile.am:145: GL_GENERATE_STDDEF_H does not appear in AM_CONDITIONAL

[Kevin R. Bulgrien]

Being in a situation where it is necessary to port some Linux software to a 
very old UNIX (OpenServer 5.0.7), and recognizing this platform is sadly
dated and not supported, some advice is sought regarding a challenge with
trying to use gnulib to add some missing things.

As the vendor supplied autotools were dated (automake 1.7.1 and its friends),
it seemed necessary/prudent to build newer autotools ... successfully done up
to automake 1.9.6, autoconf 2.6.5, and m4 1.4.9.  It is not clear how viable
to try to progress past this point, but, it may be possible that with these
tools one could move further ahead, but, at the time, various package build
failures using the old tool chain required backing off this far from current.
(Newb alert:  This is my first attempt at an endeavor such as this.)

This is what I did (for starters) with a recent gnulib pulled from git
a few weeks ago:

$ gnulib-tool --import getopt-posix

Suggested configure.ac and Makefile.am were made, then, with the old tools,

Without the newer tools, a recent gnulib pulled from git a few weeks ago
now said:

  $ autoreconf
  autoreconf: `aclocal.m4' is unchanged
  autoheader: `config.h.in' is unchanged
  lib/Makefile.am:145: GL_GENERATE_STDDEF_H does not appear in AM_CONDITIONAL
  lib/Makefile.am:26: require Automake 1.9.6, but have 1.7.1
  autoreconf: automake failed with exit status: 1

With the new tools, progress:

  $ autoreconf
  lib/Makefile.am:145: GL_GENERATE_STDDEF_H does not appear in AM_CONDITIONAL
  autoreconf: automake failed with exit status: 1

Not knowing how to resolve this last error, I found:

  http://lists.gnu.org/archive/html/bug-gnulib/2011-05/msg00467.html
  http://lists.gnu.org/archive/html/bug-automake/2011-05/msg00023.html
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=8718

Here I see what seems to be an issue that was marked as WONTFIX, and, in the
debbugs.gnu.org thread, a potential workaround posted by Bruno Haible, but,
finding myself unsure how to implement the workaround, it seems reasonable
to try for advice here.

In the aforementioned message threads, can be found mention that this issue is
related to a gnulib "conditional dependencies" feature, and, looking at git,
the feature appears to have been developed between 2011-05-03 through
2011-05-28 based on commits by Bruno Haible.

It seems somewhat plausible to back down my git checkout to just prior to this
date and make an attempt to use an old gnulib, and that is all the idea I have
to try at the moment considering my minimal experience with the autotools.

In any case, any advice given on how to proceed in this case is much certainly
appreciated.  Apologies in advance that this regard such dated resources.
Sadly, in the near term, it is difficult to shrug off these old rusty
chains in a timely manner.

Does the approach of backing down to an old gnulib seem best, or am I perhaps
not so far from a plausible workaround with a top-of-tree gnulib?  (Considering
the age of the target system, certainly working with an old gnulib can hardly
be much of a stumbling block.)

-- 
Kevin R. Bulgrien

Re: lib/Makefile.am:145: GL_GENERATE_STDDEF_H does not appear in AM_CONDITIONAL

[Bruno Haible]

Kevin R. Bulgrien wrote:

> Not knowing how to resolve this last error, I found:
> 
>   http://lists.gnu.org/archive/html/bug-gnulib/2011-05/msg00467.html
>   http://lists.gnu.org/archive/html/bug-automake/2011-05/msg00023.html
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=8718
> 
> Here I see what seems to be an issue that was marked as WONTFIX, and, in the
> debbugs.gnu.org thread, a potential workaround posted by Bruno Haible

It is marked as "won't fix", but in this thread I wrote:

  This solves my problem. Thanks for the idea to use AC_CONFIG_COMMANDS_PRE.
  I will add this workaround to gnulib.

I don't remember why I never implemented this solution. But I believe that
the patch by Paul Eggert from a while later
  https://lists.gnu.org/archive/html/bug-gnulib/2011-10/msg00106.html
fixed this problem.

> This is what I did (for starters) with a recent gnulib pulled from git
> a few weeks ago: ...
> 
> As the vendor supplied autotools were dated (automake 1.7.1 and its friends),
> it seemed necessary/prudent to build newer autotools ... successfully done up
> to automake 1.9.6, autoconf 2.6.5, and m4 1.4.9.

Automake 1.9.6 is 13 years old!! We are not supporting such old releases any 
more.
The current gnulib/DEPENDENCIES says that the minimum you need is Automake 
1.11.1
(which is 8 years old); you should better start to use the newest Automake
1.16.1 immediately.

Note also that the GNU build tools distinguish between a development machine
and a target machine. On the development machine you create tarballs with a
configure script; on the target machine you run the configure script and 'make'.
On the development machine, you need recent Autotools; on the target machine
(your very old system) you don't need anything beyond sh, common utilities,
grep, sed, awk, and make.

> With the new tools, progress:
> 
>   $ autoreconf
>   lib/Makefile.am:145: GL_GENERATE_STDDEF_H does not appear in AM_CONDITIONAL
>   autoreconf: automake failed with exit status: 1

Can you reproduce this with the newest Automake and Autoconf? If so, please
make the tarball available so that we can analyze it.

> Does the approach of backing down to an old gnulib seem best

Not at all. We won't spend time on fixing things that cannot be reproduced with
recent gnulib and recent Automake.

Bruno