Hello,
Regarding edition of .la files when they are incorrect, yes, this is what I do
where I have write access (i.e. on my laptop), but this is not the case on
large systems, and I would not be comfortable asking an admin to change a given
line for a library at the bottom of a large dependency chain for fear of
breaking something else.
Your solution of writing a compiler wrapper to adjust options as a workaround
is a good idea. I have a similar wrapper to launch debuggers, but hadn't though
of using that approach as a libtool workaround. So I'll keep that option in
mind.
Still, a --ignore-la-files option or something similar for libtool would avoid
jumping through such hoops...
Best regards,
Yvan Fournier
________________________________________
De : Automake <automake-bounces+yvan.fournier=edf...@gnu.org> de la part de
automake-requ...@gnu.org <automake-requ...@gnu.org>
Envoyé : lundi 3 mai 2021 18:00
À : automake@gnu.org
Objet : Automake Digest, Vol 216, Issue 1
Send Automake mailing list submissions to
automake@gnu.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.gnu.org/mailman/listinfo/automake
or, via email, send a message with subject or body 'help' to
automake-requ...@gnu.org
You can reach the person managing the list at
automake-ow...@gnu.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Automake digest..."
Today's Topics:
1. Re: Future plans for Autotools (FOURNIER Yvan)
2. Re: Future plans for Autotools (Thomas Jahns)
----------------------------------------------------------------------
Message: 1
Date: Sun, 2 May 2021 17:49:33 +0000
From: FOURNIER Yvan <yvan.fourn...@edf.fr>
To: "automake@gnu.org" <automake@gnu.org>
Subject: Re: Future plans for Autotools
Message-ID: <1619977772999.94...@edf.fr>
Content-Type: text/plain; charset="iso-8859-1"
Hello,
Sorry for reacting a bit late to the January discussion about Autotool's future.
As a longtime user of the GNU Autotools for a large computational dynamics code
(code_saturne.org), here are a few hopefully constructive remarks about our
experience so far.
Sorry if my post is a bit long.
Most-liked aspects
-------------------------
- excellent cross-compilation support
- `configure --help` is very practical (CMake's equivalent is much messier, and
one of the reasons we did not switch)
- "almost" no dependencies on the target system except for a shell and Make.
This is especially useful on compute clusters which may have a minimal system.
- pretty good documentation
- it is very easy to find the configure options from a previous run at the top
of the config.log and config.status files, and then copy/past them so as to
generate an new build in a clean (empty) build directory. This is IMHO a strong
point compared to CMake, whose caching aspects can lead to complex issues, and
where regenerating a configuration command is not so easy.
Some Inconsistencies
-----------------------------
While configure/automale/libtool seem to be designed to work together, there
are a few inconsistencies :
- autoconf can generate multiple configuration files (for example
public/private header files), but automake assumes only one is used. I don't
remeber the details, but had to work around this 15 years ago to re-implement
the generation of a "light" external config.h in m4 and shell instead of using
built-n features due to this.
- Though one of the initial advantages of the GNU autotools was that a shell
and Make were sufficient on the target system, GNOME devs started breking that
with pkg-config, and now it seems resorting to that has become ubiquitous in
many autotools-based projects, so the dependency list has increased in
practice. Tools like autopoint also add complexity, as they are available on
some systems not on others.
- Our code is a mix of Fortran and C, with a bit of C++. Automake still deos
not support Fortran 90-type module dependencies, so we have to manage manual
dependencies in one of our Makefile.am's. More modern systems handle Fortran
(not quite the latest fad) much better.
- When using Gettext, "make update-po" modifies code in the source tree based
on a command in the build tree. When using separate source and build trees,
this is somewhat surprising, so having an equivalent command (not based on the
generated Makefiles) for the source tree would be more consistent as regards
source/build separation.
Major issue
---------------
Worse, autoconf does not take advantage of libtool's ".am" files to find
dependencies, while libtool does not allow bypassing these files. This means
that:
- we need to manually search for dependencies anyways
- but when it encounters .la files with incorrect .la files, libtool adds the
incorrect dependencies or options, with no workaround
So while .la files were probably a great idea in the early days of the GNU
project, they are of limited use now (especially since tools not based on
autotools do not provide them), and are a source of issues.
I have encountered several examples of this :
- .la files broken by Linux packaging: in the case of an MPI library built with
various dependencies, some of which are loaded dynamically as plugins, and not
installed by default with the minimal packages, the .la file lists some
non-installed (and non-needed) libraries as dependencies.
- incorrect options: recently, trying to use the Cray compiler toolchain on an
Arm64 machine, some low-level dependency related to back-end gcc libraries
added a "-pthread" option in the link line, while the top-level compiler used a
different syntax and did not accept that.
Although I have quite a bit of experience working around libtool's quirlks, I
was simply not able to build my code using that toolchain on that machine.
Total failure just before the finish line...
-I have had a similar issue generating shared libraries with with CUDA, though
in that case I could work around this by duplicating and patching the generated
libtool file, and adding a Makefile.am rule for .cu files. But I have never
seen another project optionally using CUDA which bothered doing this rather
than using another build system (ie. if there is a better solution, I did not
find it).
Libtool has many nice aspects, such as handling of versioning, rpaths, and
more, but its
insistence that it is always right, even in situations where it produces
unusable commands can really be a showstopper.
I am seriously contemplating removing libtool (while keeping autoconf/automake)
due to the blocking "no option to ignore .la files even when incorrect" issue,
as I have no other workaround. This will probably require quite a bit of extra
work, but is better than hitting a wall.
Comparison with other tools
-------------------------------------
Frequently installing dependencies with CMake, and even adding a call to CMake
to detect configuration options in our Autotools scripts to detect a
CMake-based library's configuration, I can attest that CMake is not always fun
to work with either when things go wrong or less than perfect configuration
files are used. So CMake can also be quite frustrating.
I have not looked at Meson much so far. In the short term, the fact that it
requires ninja, which is not yet ubiquitous on HPC systems, makes things a bit
less comfortable in our case, but the fact that is uses Python is an advantage
(Python is so prevalent in scientific software that just about all users and
developers in our community need to learn at least some rudiments anyways,
which is not true of m4, Makefiles, or cmake). So it could become a serious
contender (whereas the other Python-based contenders such as scons and Waf seem
to have less momentum).
General impression
---------------------------
while we have a large investment (several thousand lines of local .m4, plus
Makefile.am) in packaging with the GNU autotools, two opposite options which
would make us happy would be to :
- either better integrate them, so as to remove the
inconstencies/interoperability issues between packages
- or make them more modular, and better able to work independently. This is
probably more of a documentation / example code issue than an actual
development issue.
As an application software developer and maintainer, I fully appreciate the
hard and painstaking work that goes into the autotools, and know how difficult
it can be to build a working and realistic roadmap and balance development and
support efforts, and thank to autotools team for keeping these tools in a
usable state.
Just hoping these comments are useful.
Best regards,
Yvan Fournier
Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à
l'intention exclusive des destinataires et les informations qui y figurent sont
strictement confidentielles. Toute utilisation de ce Message non conforme à sa
destination, toute diffusion ou toute publication totale ou partielle, est
interdite sauf autorisation expresse.
Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le
copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si
vous avez reçu ce Message par erreur, merci de le supprimer de votre système,
ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support
que ce soit. Nous vous remercions également d'en avertir immédiatement
l'expéditeur par retour du message.
Il est impossible de garantir que les communications par messagerie
électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
erreur ou virus.
____________________________________________________
This message and any attachments (the 'Message') are intended solely for the
addressees. The information contained in this Message is confidential. Any use
of information contained in this Message not in accord with its purpose, any
dissemination or disclosure, either whole or partial, is prohibited except
formal approval.
If you are not the addressee, you may not copy, forward, disclose or use any
part of it. If you have received this message in error, please delete it and
all copies from your system and notify the sender immediately by return message.
E-mail communication cannot be guaranteed to be timely secure, error or
virus-free.
------------------------------
Message: 2
Date: Mon, 3 May 2021 11:33:28 +0200
From: Thomas Jahns <ja...@dkrz.de>
To: automake@gnu.org
Subject: Re: Future plans for Autotools
Message-ID: <90fba7d1-9219-46a6-da50-e83b3c2b5...@dkrz.de>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Hello,
some comments with a similar HPC background:
On 2021-05-02 19:49, FOURNIER Yvan wrote:
[...]
> - it is very easy to find the configure options from a previous run at the
> top of the config.log and config.status files, and then copy/past them so as
> to generate an new build in a clean (empty) build directory. This is IMHO a
> strong point compared to CMake, whose caching aspects can lead to complex
> issues, and where regenerating a configuration command is not so easy.
This is indeed valuable, but could be improved by adding the appropriate quotes.
[...]
> - autoconf can generate multiple configuration files (for example
> public/private header files), but automake assumes only one is used. I don't
> remeber the details, but had to work around this 15 years ago to re-implement
> the generation of a "light" external config.h in m4 and shell instead of
> using built-n features due to this.
We also found that we could not install config.h and use a Perl script to insert
config.h macros into the installed headers.
> - Our code is a mix of Fortran and C, with a bit of C++. Automake still deos
> not support Fortran 90-type module dependencies, so we have to manage manual
> dependencies in one of our Makefile.am's. More modern systems handle Fortran
> (not quite the latest fad) much better.
At our site, there are a number of scripts in use that generate the Fortran
module/file dependencies on demand. For those not knowing Fortran: source files
are compiled to both, .mod files for use in runs of the compiler and .o files
for the later link step. The .mod files can follow a number of conventions in
respect to suffix and upper/lower case basename and suffix.
[...]
> - .la files broken by Linux packaging: in the case of an MPI library built
> with various dependencies, some of which are loaded dynamically as plugins,
> and not installed by default with the minimal packages, the .la file lists
> some non-installed (and non-needed) libraries as dependencies.
I found that in general it's advisable to look through .la files and edit them
if needed.
> - incorrect options: recently, trying to use the Cray compiler toolchain on
> an Arm64 machine, some low-level dependency related to back-end gcc libraries
> added a "-pthread" option in the link line, while the top-level compiler used
> a different syntax and did not accept that.
We encounter the same problem with the NAG Fortran compiler (equivalent options
are -Wc,-pthread -Wl,-pthread). The simplest solution was a wrapper around to
the compiler to patch up the options, another colleague patched ltmain.sh to
adjust appropriately.
> Although I have quite a bit of experience working around libtool's quirlks, I
> was simply not able to build my code using that toolchain on that machine.
> Total failure just before the finish line...
>
>
> -I have had a similar issue generating shared libraries with with CUDA,
> though in that case I could work around this by duplicating and patching the
> generated libtool file, and adding a Makefile.am rule for .cu files. But I
> have never seen another project optionally using CUDA which bothered doing
> this rather than using another build system (ie. if there is a better
> solution, I did not find it).
Since CUDA depends on the installed kernel module, it's also giving us headaches
regarding stability of installed libraries like MPI. There is support for
optional .cu files in our highest level software, I'll have to look up what the
responsible colleague did that might have eluded you. We already depend on a
build environment with the PGI/NVIDIA compiler for OpenACC when GPU support is
needed and that adds everything for cuda below the layer visible to libtool, we
might not be seeing your specific issue.
[...]
> Libtool has many nice aspects, such as handling of versioning, rpaths, and
> more, but its
>
> insistence that it is always right, even in situations where it produces
> unusable commands can really be a showstopper.
I can second that, we now use 7 patch sets for ltmain.sh to address issues
regarding compilers not matching the expectations of libtool.
> I have not looked at Meson much so far. In the short term, the fact that it
> requires ninja, which is not yet ubiquitous on HPC systems, makes things a
> bit less comfortable in our case, but the fact that is uses Python is an
> advantage (Python is so prevalent in scientific software that just about all
> users and developers in our community need to learn at least some rudiments
> anyways, which is not true of m4, Makefiles, or cmake). So it could become a
> serious contender (whereas the other Python-based contenders such as scons
> and Waf seem to have less momentum).
I found scons to be sufficiently complex to be impenetrable to a number of
scientists, who otherwise knew and used Python.
Regards, Thomas
--
Thomas Jahns
HPC-Gruppe
Abteilung Anwendungssoftware
Deutsches Klimarechenzentrum GmbH
Bundesstraße 45a • D-20146 Hamburg • Germany
Phone: +49 40 460094-151
Fax: +49 40 460094-270
Email: Thomas Jahns <ja...@dkrz.de>
URL: www.dkrz.de
Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5348 bytes
Desc: S/MIME Cryptographic Signature
URL:
<https://lists.gnu.org/archive/html/automake/attachments/20210503/0d8057a1/attachment.bin>
------------------------------
Subject: Digest Footer
_______________________________________________
Automake mailing list
Automake@gnu.org
https://lists.gnu.org/mailman/listinfo/automake
------------------------------
End of Automake Digest, Vol 216, Issue 1
****************************************
Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à
l'intention exclusive des destinataires et les informations qui y figurent sont
strictement confidentielles. Toute utilisation de ce Message non conforme à sa
destination, toute diffusion ou toute publication totale ou partielle, est
interdite sauf autorisation expresse.
Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le
copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si
vous avez reçu ce Message par erreur, merci de le supprimer de votre système,
ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support
que ce soit. Nous vous remercions également d'en avertir immédiatement
l'expéditeur par retour du message.
Il est impossible de garantir que les communications par messagerie
électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
erreur ou virus.
____________________________________________________
This message and any attachments (the 'Message') are intended solely for the
addressees. The information contained in this Message is confidential. Any use
of information contained in this Message not in accord with its purpose, any
dissemination or disclosure, either whole or partial, is prohibited except
formal approval.
If you are not the addressee, you may not copy, forward, disclose or use any
part of it. If you have received this message in error, please delete it and
all copies from your system and notify the sender immediately by return message.
E-mail communication cannot be guaranteed to be timely secure, error or
virus-free.
