Very slow links: > 1 hour to link single executable - can we help?
(After looking at the past few months of this list, I didn't see any
relevant posts, but if I just overlooked them, please let me know.)
With more recent libtool versions I've noticed suprisingly slow
libtool link times. After some investigation, I discovered some gnome
related mail discussing the issue that suggested that libtool was
having problems with duplicate libs listed on the link line.
To try and fix this, I used the adapted snippet from gnome-config
listed below to safely remove duplicates from the link line. This
worked, taking the compiles down from 20 minutes to 10 (this is on a
1GHz Athlon), and this was
However, since we're about to need to be able to lt_dlopen/dlsym most,
if not all of our libraries, I found that I needed to go back through
and make sure that all of our libs were fully inter-linked. This
meant more libfoo.la links to each lib's libfoo_la_LIBADD.
After finishing all that, the compile times have jumped way past 20
minutes, to at least more than 90 minutes (as I compose this, the
compile still hasn't finished) -- It looks like duplicates are being
pulled in through the .la files themselves, and I there's no way for
me to clean that up before invoking libtool as I was with the
gnome-config hack.
If I edit our ./libtool file, and add an "set +x", I can see libtool
running some kind of recursive dependency analysis which over hundreds
of iterations builds the deplibs (I think) variable up to pages and
pages of duplicate libs.
If I strace the sh it's runing, I can see that it's trying to stat the
same library, say libm, in many locations, only to turn around and do
that all again. Right now it has taken over an hour just trying to
link the final executable, and it's still not finished.
My questions:
1) Is this expected behavior, or are we doing something wrong?
2) If this is expected, are there plans to fix it?
3) If there are plans, what can we do to help? :>
Oh, and right now, we're using libtool 1.4.2.
Thanks
#!/bin/sh
# This script removes any duplicate occurences of -lfoo in its args,
# but preserves the right ordering (last-duplicate-wins) for the
# linker. Code originally from gnome-config.
all_flags="$@"
# Straight out any possible duplicates, but be careful to
# get `-lfoo -lbar -lbaz' for `-lfoo -lbaz -lbar -lbaz'
other_flags=
lib_L_flags=
rev_libs=
for i in $all_flags; do
case "$i" in
# a library, save it for later, in reverse order
-l*|*.la) rev_libs="$i $rev_libs" ;;
-L*|-R*)
if $libs_L; then
case " $lib_L_flags " in
*\ $i\ *) ;;# already there
*) lib_L_flags="$lib_L_flags $i" ;; # add it to output
esac
fi;;
*)
case " $other_flags " in
*\ $i\ *) ;;# already there
*) other_flags="$other_flags $i" ;; # add it to output
esac ;;
esac
done
ord_libs=
if $libs_l; then
for i in $rev_libs; do
case " $ord_libs " in
*\ $i\ *) ;;# already there
*) ord_libs="$i $ord_libs" ;; # add it to output in reverse order
esac
done
fi
echo $other_flags $lib_L_flags $ord_libs
exit 0
--
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD
___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool
A hack we've been using -- is this OK, or likely to fail?
I meant to mention this in my previous mail, but forgot. Because we have developers using distributions for which it's not easy to upgrade to the latest libtool, instead of using libfoo_la_LIBADD = ../somedir/libbar.la to indicate inter-lib dependencies, we've been using libfoo_la_LIBADD = -L../somedir -L../somedir/.libs -lbar This has made me a little nervous, so I wanted to check to see if this is likely to work OK, either under 1.4.2 or prior versions, or if this could be expected to cause trouble. Thanks -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: duplicate dependencies [was: RE: Very slow links: > 1 hour tolin k single executable - can we help?]
"Boehne, Robert" <[EMAIL PROTECTED]> writes: > This is a known problem that results from supporting multiple > dependent archive libraries. In some cases libtool can't strip the > redundant -lfoo -lbar in dependencies. I would propose that libtool > would strip all duplicate dependencies (as it used to do) unless a > command line flag is given. I did pose this question a few weeks > ago but didn't get any response. Time permitting, I'll post a patch > that would require the KDE developers (and others in this situation) > to add a flag that preserves all of the dependencies when linking. > In my project, compile times go from 6 hours to infinity. ;) So > I've made my own ltmain.sh that won't preserve any duplicate > dependencies. As soon as I can find the time I'll work this into a > patch. In the mean time, anyone care to suggest what the flag > should be? --preserve-dup-deps ??? As another data point. I we installed your ltmain.sh in gnucash and link times on a 1Ghz athlon went from two hours to ten minutes, and everything still seems to work OK. Thanks. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
What about adding lt_dlopen_interface(lib, number)?
How would people feel about the addition of a new function
lt_dlhandle lt_dlopen_interface(const char *name, int interface_number);
which would return a handle to a library satisfying the given
interface_number or NULL if no such library is available in the search
path.
Without this, there's no way to (portably) make sure you get the
actual library you need. You just get first-available, which might be
the wrong one.
Further, the job this function is intended to handle is not one that
the calling code could easily implement, since only libtool is
supposed to know the architecture specific details regarding a given
library's filename, and since libtool should probably be solely
responsible for deciding which library versions satisfy which
interfaces.
So you would say:
lib = lt_dlopen_interface("libsomething" 4);
Trying to solve this problem outside libtool seems likely to involve
fairly awkward library filesystem placement, and LD_LIBRARY_PATH
tricks.
If people think this is a good idea, I'd be happy to help write the
code.
Thanks
--
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD
___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool
Re: use of libtool for linking executables - rpath problem
Bruno Haible <[EMAIL PROTECTED]> writes: > 3) Introduce a libintl-config script that sets outputs the right -L and >-rpath option. This may or may not help you if you're linking with *any* other tools that are installed in /usr/lib and their foo-config scripts output -L/usr/lib. In that case you *cannot* in general be assured of linking against a locally installed version of libintl whenever you also have another version (including the development .so links) installed in /usr/lib. The one in /usr/lib, if the other package's -L/usr/lib comes first, will take prececence. The only clean solution I could see to this would be for gcc to support a --push-state --pop-state (or similar) set of args so that a given config script could output: --push-state -L/usr/lib -lgnome-print -L... -l... --pop-state and not have it's -L args screw up later flags on the compile line. But though I've had some agreement from relevant developers, I wouldn't expect to see this any time soon. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: use of libtool for linking executables - rpath problem
[EMAIL PROTECTED] (Paul Jarc) writes: > Uh? AIUI, the basename of the shared library is embedded in the > executable, but not the full path. The library is searched for first > in the -rpath directories (also embedded in the executable) and then > in the global directories such as /usr/lib. The problem I'm talking about is if Makefile.am's use LDFLAGS = `gnome-config --libs foo` `foo-config --libs bar` then if you've got a normal gnome-dev package installed, such that it's libs are in /usr/lib, it will (or at least it used to) put -L/usr/lib into LDFLAGS ahead of whatever foo-config specifies. Now if the system you're on has foo version 1.1 installed in /usr but you want to develop for foo version 2.0, even if you install foo 2.0 into ~/opt/foo-2.0, you can't develop with it because gnome-config's -L/usr/lib will force you to use the foo 1.1 version in /usr/lib. Now you can get around this by changing the order above, but that won't scale, and breaks if the situation wrt foo and gnome were reversed. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
libtool doesn't allow "make DESTDIR=`pwd`/foo install" anymore.
If you build a package with --prefix=/usr and that package has interdependencies among it's shared libraries (like guile and heimdal), libtool will no longer allow you to install to a temporary directory via make DESTDIR=`pwd`/foo install libtool 1.4 allowed this, but as of at least 1.4.2, it doesn't. This makes it very difficult, if not impossible to package programs using libtool for systems like debian that require creating a local install from which the package is generated. See http://bugs.debian.org/libtool, bugs 57087 and 98342. I believe this may have been discussed here at some point in the past, but I just wanted to check and make sure that the current developers were still aware of the problem, and if so, see if there were plans to fix it. I also wanted to see if anyone had suggestions for a fix, even a temporary one. There's a patch in the debian package that seems to fix the problem for some people, but still doesn't work for other package like guile and heimdal. Thanks -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: libtool doesn't allow "make DESTDIR=`pwd`/foo install" anymore.
Paul E Johnson <[EMAIL PROTECTED]> writes: > I do now know what patch is required to make libtool-1.4.2 work as > it is currently released, I've been asking about. My understanding > is that the Debian packager of libtool has a patch which fixes the > problem. Hmm, well the latest Debian libtool claimed to have such a patch, but at least for guile and heimdal, it didn't fix the problem, though perhaps these packages aren't using the automake variables the way that libtool would prefer. For libguile, we have: libguile_la_DEPENDENCIES = @LIBLOBJS@ libguile_la_LIBADD = @LIBLOBJS@ $(LIBLTDL) $(THREAD_LIBS_LOCAL) libguile_la_LDFLAGS = -version-info @LIBGUILE_INTERFACE_CURRENT@:@LIBGUILE_INTERFACE_REVISION@:@LIBGUILE_INTERFACE_AGE@ -export-dynamic -no-undefined where $(THREAD_LIBS_LOCAL) := ../qt/libqthreads.la -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: libtool doesn't allow "make DESTDIR=`pwd`/foo install" anymore.
Paul E Johnson <[EMAIL PROTECTED]> writes: > It easily could be true that tarballs or SRPMS that were built with > old versions of libtool and automake/autoconf do not rebuild with > the current libtool. I suspect this is true -- my experience has been that some of the people for whom this is still working that I've asked to tell me what ./libtool --version returns tell me that they're actually still running 1.4. In any case, it would be nice to see a test added to a libtool "make check" step for this problem (if that hasn't been done already) so that we don't regress in the future. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: What about adding lt_dlopen_interface(lib, number)?
Rob Browning <[EMAIL PROTECTED]> writes: > How would people feel about the addition of a new function > > lt_dlhandle lt_dlopen_interface(const char *name, int interface_number); So do the libtool maintainers have any opinion about this one way or the other? As it stands, Guile, and I suspect any other apps that want to use dlopened libraries more heavily, especially user created libraries that may depend on each other, really have to do something to make sure they can rely on getting the interface they require when opening a library. To me, using the infrastructure libtool already supports as suggested above, seems the most natural thing, but if that's not acceptable, then Guile will probably need to implement something parallel to that internally, perhaps using a naming convention that embeds the versioning information in the library name, or something. In any case, I just wanted to see if there's sentiment either way so we can take it into consideration as we discuss what we need to do next in Guile. Thanks -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Hack for inter-library dependency (make DESTDIR=... install)problem.
For those stuck on this problem (and presuming we're not just using libtool improperly in Guile) I've figured out a temporary hack that will allow you to build and install packages in temporary (i.e. not --prefix) directories when you have inter-library dependencies. I just had to add a spurious -L for the build directory of the depended on library whereever appropriate. i.e. libguilereadline_la_LIBADD = ../libguile/libguile.la became libguilereadline_la_LIBADD = -L`pwd`/../libguile/.libs ../libguile/libguile.la This may add an unwanted -rpath, so if someone has a better suggestion, I'd be happy to hear it. Hope this helps. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: Hack for inter-library dependency (make DESTDIR=... install)problem.
Brian May <[EMAIL PROTECTED]> writes: > While this would work, it has the (IMHO) nasty side affect that any > libraries previously installed in, say /usr/lib, will take priority (I > think) over the library that was just built in the source code tree. > > It also means hacking around with the Makefiles. Agreed, but for me right now, it's better than nothing, and I needed *some* fix to allow some needed inter-lib dependencies to be included in the 1.4 packages. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Allowing guile to open a lib satisfying a given interface.
I'm still haven't heard anything about how the libtool developers feel
about the issue, but since we really need some solution to this
problem, I'm wondering we might be able to implement our own
scm_lt_dlopen(char *name, unsigned int interface) for now as follows.
The current libltdl may or may not allow this, but here's what I was
considering:
1) require that anyone creating a shared lib that's to be opened by
guile in support of a guile module install a copy of the
libfoo.la file as libfoo.la.X.Y.Z using the appropriate version
numbers X, Y, and Z.
2) write a function like scm_lt_dlopen("libfoo", 4) that will search
in the same places that libltdl does for a libfoo.la.X.Y.Z that
satisfies the interface number (i.e. "4") specified in the call
to scm_lt_dlopen using the libtool version algorithm. If such a
.la file is found, then open it explicitly via
lt_dlopen("/full/path/to/libfoo.la.X.Y.Z").
The main question is whether or not lt_dlopen really cares about the
actual name of the .la file. If it doesn't, then we may be OK.
When I get a chance, I'll probably look at the source and perform some
tests, but this seems like it might solve our problems, and if it does
work, then perhaps some variation on this approach might be acceptable
to the libtool maintainers.
Thoughts?
--
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD
___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool
Re: Hack for inter-library dependency (make DESTDIR=... install)problem.
[EMAIL PROTECTED] writes:
> Another solution (probably covered here) is to fill in the
> LIBRARY_PATH search path:
>
> for file in `find $builddir -name "*.la" -print xargs -i sed -e
>"s:libdir='\(.*\)':\1:" {}`; do
> LIBRARY_PATH=$LIBRARY_PATH:$DESTDIR$file
> done
>
> I haven't looked at the latest libtools source, but this may already
> be implemented.
Could you explain this a little further? By LIBRARY_PATH here do you
mean LD_LIBRARY_PATH (at least for Linux)?
If so, I'm not sure this will help since the relink step explicitly
mentions -L/usr/lib -lfoo when I'm actually installing into ./tmp via
"make DESTDIR=`pwd`/tmp install", and of course, foo's in pwd/tmp, not
/usr/lib yet.
What I need is for the relink step to link against the build tree
libfoo (or more likely the DESTDIR libfoo) and not worry about the
fact that the library's eventually going to be in /usr/lib...
I'm curious about your suggestion because the hack I'm using right now
is causing problems, and I'm going to have to come up with something
better. I've even thought of sed'ing the .la files in place, but I'd
love a better alternative.
Thanks
--
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD
___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool
Re: Hack for inter-library dependency (make DESTDIR=... install)problem.
[EMAIL PROTECTED] writes: > The LD_LIBRARY_PATH variable is for ELF systems to find shared > libraries at run-time. The LIBRARY_PATH variable, on the other hand, > is a gcc setting that lets it find libraries at link-time. It's a > colon-separated path that fills in a bunch of '-L' options. Other > linkers have similar extensions -- LPATH, LOPTS, etc. OK, thanks. I was confused because I knew about LD_LIBRARY_PATH, but not LIBRARY_PATH. > I've used this trick successfully in building rpm's containing > multiple, interdependent shared libraries. The spec file cooks up a > custom LIBRARY_PATH setting before invoking 'make DESTDIR=foo > install'. The name of the game in spec file writing is to minimize > the patch set required to build the package, so Makefile hackery is > discouraged. Agreed -- the same is true with Debian -- minimizing the Debian diff is a good idea, all other things being equal. In any case, thanks so much. I'll try this -- it looks like a good solution. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: Hack for inter-library dependency (make DESTDIR=... install)problem.
Rob Browning <[EMAIL PROTECTED]> writes: > Agreed -- the same is true with Debian -- minimizing the Debian diff > is a good idea, all other things being equal. In any case, thanks > so much. I'll try this -- it looks like a good solution. That appeared to work perfectly -- thanks again. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: stopping relinking under elf/linux
Jens Petersen <[EMAIL PROTECTED]> writes: > It is my understanding that relinking is not needed on elf > or Linux binary platforms. Further relinking actually gets > in the way when making binary packages, since "make install" > is usually installing into a buildroot. The easiest way I know of to fix the buildroot problem is to configure your package with --prefix set to the eventual install location, but to use gcc's LIBRARY_PATH to keep libtool from choking on the "make install" to the temp dir. i.e.: LIBRARY_PATH=$(pwd)/debian/tmp make ... install or similar. Hope this helps. -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: Trouble controlling linking if many library versions areinstalled
Steve Tell <[EMAIL PROTECTED]> writes: > In additional experimention, I found that simply removing -L/usr/lib > altogether works in this case, perhaps because of the --rpath. > "Correct" for me here is building a libguilereadline.so that depends > explicitly on the versioned name libguile.so.14 (which I guess will > be searched for at runtime) We've had a similar ongoing problem with the foo-config type scripts, i.e. "foo-config --libs". If any one of them that you're using to compile a given project produces -I/usr/include or -L/usr/lib, and if its flags appear on your compile/link line ahead of some other lib, then its impossible to build your project using a non /usr/lib install of some library that's also installed in /usr/lib. For example, if you have guile 1.4 installed in /usr/lib, and you install the guile 1.5 beta into /opt/guile-1.5, you will be unable to compile or link a gtk app against guile 1.5 if any of the gtk config scripts output -L/usr/lib and those flags appear before -lguile on the link line. On Linux and similar systems, could libtool, foo-config scripts, etc. just special case (and omit) -L/usr/lib -L/lib and -I/usr/include? Thanks -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: Buffer overruns in libltdl/ltdl.c
Julian Seward <[EMAIL PROTECTED]> writes: > I just looked at rev 1.163 in your CVS repo, and it's definition > of realloc is similarly broken (assuming my analysis below is > correct). Are you aware of this? I reported this a while ago too -- I sure hope it gets fixed -- it was causing all manner of problems here. Why does ltdl need to implement its own memory management anyway? -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
Re: Trouble controlling linking if many library versions areinstalled
Bob Friesenhahn <[EMAIL PROTECTED]> writes: > The foo-config type scripts are fraught with problems. They attempt > to be a complete solution rather than expressing only the dependencies > they directly contribute. Unfortunately, eliminating -L/usr/lib > -L/lib and -I/usr/include is not an appropriate solution because it > may be that library dependencies dictate that a library in this > directory is required prior to another library. The only solution may > be explicit linkage. I guess the eventual solution might have to be gcc args like --push-flags --pop-flags, or perhaps better, --begin-group --end-group as someone recently mentioned, but that's not going to help on all architectures... -- Rob Browning rlb @defaultvalue.org, @linuxdevel.com, and @debian.org Previously @cs.utexas.edu GPG=1C58 8B2C FB5E 3F64 EA5C 64AE 78FE E5FE F0CB A0AD ___ Libtool mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/libtool
