Avoid la files.

[Duft Markus]

Hi All!

I have a quick question about libtool: What can I break, if I fool
libtool into thinking that there are no .la files for any library on the
command line?

The background: We have a bunch of software (about 1000 libraries) which
are linked (shared) together in one executable. If manually linking it
takes about 30 seconds. If linking with libtool it takes 40 minutes
You see the problem? If I just comment out the double loop in libtool at
line (approximately) 2517 and set found=no allways, it takes only 36
seconds.

We have a completely standard environment, use gcc allways (but with
native linkers on some platforms). We're using AIX, SUN (Sparc, i386),
Linux (x86, x64_86), Windows (parity), HP (Itanium, PA-RISC).

Any ideas?

Cheers, Markus

-- 
19. - 21. Februar 2008
Salomon Automation auf der LogiMAT 2008, Neue Messe Stuttgart, Deutschland
Halle 6, Stand 527

23. - 27. Februar 2008
MoveRetail auf der EuroShop 2008 in Dusseldorf, Deutschland
Halle 6, Stand C50





Salomon Automation GmbH - Friesachstrasse 15 - A-8114 Friesach bei Graz
Sitz der Gesellschaft: Friesach bei Graz
UID-NR:ATU28654300 - Firmenbuchnummer: 49324 K
Firmenbuchgericht: Landesgericht fur Zivilrechtssachen Graz



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Avoid la files.

[Ralf Wildenhues]

Hello Markus,

Thanks for the report.

* Duft Markus wrote on Thu, Dec 13, 2007 at 10:34:16AM CET:
> 
> I have a quick question about libtool: What can I break, if I fool
> libtool into thinking that there are no .la files for any library on the
> command line?
> 
> The background: We have a bunch of software (about 1000 libraries) which
> are linked (shared) together in one executable. If manually linking it
> takes about 30 seconds. If linking with libtool it takes 40 minutes
> You see the problem? If I just comment out the double loop in libtool at
> line (approximately) 2517 and set found=no allways, it takes only 36
> seconds.

Which Libtool version?  If branch-1-5, can you try out with HEAD as
well?  Are those 1000 libraries all convenience archives?  Can you post
the link command line and the output that libtool generates from it?
Are those 40 minutes on w32 or some other system as well?

To be able to help I need to be able to reproduce the scalability issue.

Cheers,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

RE: Avoid la files.

[Duft Markus]

Hello!

W32 was the fastest with 40 minutes, a x86_64 linux that i tried as well
took about 50 minutes before we canceled ;o(

I'm not sure if i can easily put in HEAD, we use 1.5.24. Would it be
enough to get HEAD built somewhere, and bootstrap only the last
component with it (leave all libraries untouched)? Maybe we can try on a
linux box, w32 won't work, since i don't have my patches ported for HEAD
;o(

I attached both the libtool call and the linker call from my w32 box, if
you wanr i can organize linux output as well.

There is not a single convenience library in play, since our tool which
handles these things (Confix, http://www.sf.net/projects/confix) is
still not able to produce those things (arg I know... We'd really
need it for our command lines)

The problem i think this command triggers is, that libtool looks at some
.la files most libraries depend on a lot of times. Maybe caching la
lookup results would be a solution, although i cannot think of how to do
this...

P.S.: the attached output has the .la searching commented out.. If you'd
need the outher output, you will have to wait a hour or two ;o) We have
two executables, both link looots of libraries, i attached both's
output.

P.P.S.: we recently changed Confix (see above) to include all dependent
libraries on the command line, because there are some weird implicit
dependencies around, which didn't get resolved otherwise... Arg - again

P.P.P.S.: please ignore the warnings in the output, they're fixed
already ;o)

And thanks for the fast reaction... ;o)

Cheers, Markus

Ralf Wildenhues <> wrote:
> Hello Markus,
> 
> Thanks for the report.
> 
> * Duft Markus wrote on Thu, Dec 13, 2007 at 10:34:16AM CET:
>> 
>> I have a quick question about libtool: What can I break, if I fool
>> libtool into thinking that there are no .la files for any library on
>> the command line? 
>> 
>> The background: We have a bunch of software (about 1000 libraries)
>> which are linked (shared) together in one executable. If manually
>> linking it takes about 30 seconds. If linking with libtool it takes
>> 40 minutes You see the problem? If I just comment out the double
>> loop in libtool at line (approximately) 2517 and set found=no
>> allways, it takes only 36 seconds.
> 
> Which Libtool version?  If branch-1-5, can you try out with HEAD as
> well?  Are those 1000 libraries all convenience archives?  Can you
> post the link command line and the output that libtool generates from
> it? Are those 40 minutes on w32 or some other system as well?
> 
> To be able to help I need to be able to reproduce the scalability
> issue. 
> 
> Cheers,
> Ralf
> 
> 
> ___
> http://lists.gnu.org/mailman/listinfo/libtool


-- 
19. - 21. Februar 2008
Salomon Automation auf der LogiMAT 2008, Neue Messe Stuttgart, Deutschland
Halle 6, Stand 527

23. - 27. Februar 2008
MoveRetail auf der EuroShop 2008 in Dusseldorf, Deutschland
Halle 6, Stand C50





Salomon Automation GmbH - Friesachstrasse 15 - A-8114 Friesach bei Graz
Sitz der Gesellschaft: Friesach bei Graz
UID-NR:ATU28654300 - Firmenbuchnummer: 49324 K
Firmenbuchgericht: Landesgericht fur Zivilrechtssachen Graz



linkout.tgz
Description: linkout.tgz
___
http://lists.gnu.org/mailman/listinfo/libtool

Debugging subversion compile failure for 64bit platform.

[[EMAIL PROTECTED]]

Hi folks,

I have been trying to compile subversion on Sun Solaris 10 Update 4, with
Sun Studio 12, for 64 bit Sparc.

The compilation for subversion as such works, almost out of the box, aside
from some minor linking quirks, which can be fixed with the setting of
LDFLAGS and others etc, pp.

However the compilation of the swig-* components fails, as libtool is
dropping the -m64 flag somewhere.
I have produced a 4607 lines long libtool --debug output for the 
` gmake DESTDIR=/tmp/build/svn/staging swig-py ' [The output for gmake ...
javahl is about double the size], which results in:

ld: fatal: file
../../../../subversion/bindings/swig/python/libsvn_swig_py/.libs/libsvn_swig
_py-1.so: wrong ELF class: ELFCLASS64
ld: fatal: File processing errors. No output written to .libs/_core.so

Prequisite to this is:

export TERM=vt100
export CC=cc
export CXX=CC
export CFLAGS="-fast -s -xarch=sparcvis2 -m64 -mt"
export CXXFLAGS=${CFLAGS}
export LDFLAGS='-s -L/opt/baw/lib -R/opt/baw/lib -L/usr/sfw/lib/64
-R/usr/sfw/lib/64 -L/usr/lib/64 -R/usr/lib/64 -L/lib/64 -R/lib/64'
export LD_LIBRARY_FLAGS=

You can find the debug output here:
http://brainsware.org/jmcg/subversion_libtool_debug.txt .
I am thankful for every hint or clue I can get on this.

Thank you in advance.

So long,
Igor


mail2web LIVE – Free email based on Microsoft® Exchange technology -
http://link.mail2web.com/LIVE




___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Debugging subversion compile failure for 64bit platform.

[Ralf Wildenhues]

Hello Igor,

* [EMAIL PROTECTED] wrote on Thu, Dec 13, 2007 at 03:25:03PM CET:
> 
> However the compilation of the swig-* components fails, as libtool is
> dropping the -m64 flag somewhere.

Which `./libtool --version'?

Typically you can work around this using
  ./configure CC='cc -m64 -xarch=...' CXX='CC -m64 -xarch=...'

but AFAIK all such instances are fixed in 1.5.24.

> I have produced a 4607 lines long libtool --debug output for the 
> ` gmake DESTDIR=/tmp/build/svn/staging swig-py ' [The output for gmake ...
> javahl is about double the size], which results in:
> 
> ld: fatal: file
> ../../../../subversion/bindings/swig/python/libsvn_swig_py/.libs/libsvn_swig
> _py-1.so: wrong ELF class: ELFCLASS64
> ld: fatal: File processing errors. No output written to .libs/_core.so

If that's from 1.5.24, then I can take a look.

Cheers,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Debugging subversion compile failure for 64bit platform.

[[EMAIL PROTECTED]]

Hi Ralf,
>  Which `./libtool --version'?
ltmain.sh (GNU libtool) 1.5.2 (1.1220.2.60 2004/01/25 12:25:08)


> Typically you can work around this using
>  ./configure CC='cc -m64 -xarch=...' CXX='CC -m64 -xarch=...'
>
>but AFAIK all such instances are fixed in 1.5.24.

Right now I'm trying to work around with what I found at
http://lists.gnu.org/archive/html/libtool/2007-12/msg00037.html
(OBJECT_MODE=64), if that doesn't work, I'll try your workaround.

>Cheers,
>Ralf

Thank you for the fast reply,

So long,
Igor



mail2web LIVE – Free email based on Microsoft® Exchange technology -
http://link.mail2web.com/LIVE




___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Avoid la files.

[Ralf Wildenhues]

* Duft Markus wrote on Thu, Dec 13, 2007 at 11:27:34AM CET:
> 
> I'm not sure if i can easily put in HEAD, we use 1.5.24. Would it be
> enough to get HEAD built somewhere, and bootstrap only the last
> component with it (leave all libraries untouched)?

Yes.

> Maybe we can try on a linux box, w32 won't work, since i don't have my
> patches ported for HEAD ;o(

Yes, that should be sufficient.  I can't tell you yet whether it would
help, though, as I'm not yet sure about the actual problem cause.

> I attached both the libtool call and the linker call from my w32 box, if
> you wanr i can organize linux output as well.

Thanks.

So I gather all of those libraries are installed shared libraries that
come from a third-party package?  And many of those libraries themselves
depend on other libraries?

> There is not a single convenience library in play, since our tool which
> handles these things (Confix, http://www.sf.net/projects/confix) is
> still not able to produce those things (arg I know... We'd really
> need it for our command lines)

I still haven't understood whether this is actually a problem in your
setup/Config/your package, and you want libtool to work around those
limitations, or you *really* need five hundred different shared
libraries and a program that links against all of them.

I mean, wow, have you ever considered that the runtime linker overhead
for this kind of setup can be excessive?  Just symbol searching is bound
to take a looong time, and that's even on sane systems like GNU/Linux.

Be kindly advised to consider some general guidance how to create shared
(ELF) libraries: .
Note also that *fewer* shared libraries is a good thing for the reasons
mentioned therein.
 
However, if all those libraries should instead be convenience archives,
because they should just be linked together into one (or a few) big
shared libraries, then you can avoid the libtool performance problems by
simply making those libraries convenience archives.

> The problem i think this command triggers is, that libtool looks at some
> .la files most libraries depend on a lot of times. Maybe caching la
> lookup results would be a solution, although i cannot think of how to do
> this...

Hmm, then I may need to look at the full output.  But please don't send
it just yet, please just answer the questions above.  Don't hesitate to
ask if there is something you don't understand, but please be precise in
your answers.

Thanks,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Debugging subversion compile failure for 64bit platform.

[Ralf Wildenhues]

* [EMAIL PROTECTED] wrote on Thu, Dec 13, 2007 at 04:39:44PM CET:
> 
> >  Which `./libtool --version'?
> ltmain.sh (GNU libtool) 1.5.2 (1.1220.2.60 2004/01/25 12:25:08)

That's way old.  Please don't report bugs against such old versions,
rather, if that libtool was generated by subversion's configure, then
please complain to the subversion people that they use an up to date
libtool version.

> > Typically you can work around this using
> >  ./configure CC='cc -m64 -xarch=...' CXX='CC -m64 -xarch=...'
> >
> >but AFAIK all such instances are fixed in 1.5.24.
> 
> Right now I'm trying to work around with what I found at
> http://lists.gnu.org/archive/html/libtool/2007-12/msg00037.html
> (OBJECT_MODE=64), if that doesn't work, I'll try your workaround.

OBJECT_MODE will not help you for the bug you reported.

Cheers,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Avoid la files.

[Bob Friesenhahn]

On Thu, 13 Dec 2007, Ralf Wildenhues wrote:



Maybe we can try on a linux box, w32 won't work, since i don't have my
patches ported for HEAD ;o(


Yes, that should be sufficient.  I can't tell you yet whether it would
help, though, as I'm not yet sure about the actual problem cause.


Something that many people are not aware of is that 'dtrace' (as found 
in Sun's Solaris 10 and Apple's OS X Leopard) may be used to 
investigate performance problems in libtool.  I mention this because I 
have never seen it mentioned on any autotools list before.  Carefully 
crafted dtrace scripts can identify which programs are being executed, 
how many times they are executed, how long the execution took, the 
arguments which produced the longest run times, and which program was 
responsible for executing them.  Virtually anything may be analyzed 
using 'dtrace'.


I have not had any spare time for months now, but I did try running 
dtrace on a shell script I have here and I quickly learned that the 
shell was executing some of its startup ("dot") files and so some 
shell code was being needlessly executed over and over.  Something 
which had not been obvious to me became immediately obvious due to 
using dtrace.


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Mac OS X 10.2.8 HEAD test failures.

[Peter O'Gorman]

Ralf Wildenhues wrote:
> Hi Peter,
> 
> * Peter O'Gorman wrote on Tue, Dec 11, 2007 at 05:11:47PM CET:
>> Ralf Wildenhues wrote:
>>
>>> That's a bug, thanks for catching.  Does it work if _LT_CHECK_BUILDDIR
>>> is only m4_require'd?
>>>
>>> I assume if that's fixed, there will still be more issues.
>>>
>> It works a bit better, now tests fail saying "Autoconf version 2.58 or
>> higher is required for this script" which is far more reasonable.
>>
>> I am not sure what to do in these cases, sure the tools are old, but
>> people will download libtool-2.x and run ./configure; make; make check,
>> and it will fail simply because they have older automake & autoconf.
>>
>> Should we print a warning at make check time warning users that their
>> versions of automake and autoconf are too old to run the testsuite?
> 
> For the tests for which it is ok that older autotools are insufficient,
> we should just skip the individual test.  I haven't had a chance to look
> at the tests to see which ones should work and which ones shouldn't.
> In any case the old-iface ones should.

Hah, I just looked to see where the 2.58 requirement was coming from ...
it is, of course LT_INIT. Any autotest test that uses autoconf will fail
with older autoconf (including old-iface).

Peter
-- 
Peter O'Gorman
http://pogma.com


___
Bug-libtool mailing list
[EMAIL PROTECTED]
http://lists.gnu.org/mailman/listinfo/bug-libtool

Re: Debugging subversion compile failure for 64bit platform.

[[EMAIL PROTECTED]]

> > >  Which `./libtool --version'?
> > ltmain.sh (GNU libtool) 1.5.2 (1.1220.2.60 2004/01/25 12:25:08)
> 
> That's way old.  Please don't report bugs against such old versions,
> rather, if that libtool was generated by subversion's configure, then
> please complain to the subversion people that they use an up to date
> libtool version.

That's the version Solaris ships. I'll try to get a newer version installed
before making any more noise here.

> > > Typically you can work around this using
> > >  ./configure CC='cc -m64 -xarch=...' CXX='CC -m64 -xarch=...'
> > >
> > >but AFAIK all such instances are fixed in 1.5.24.

This work around did not work, btw.
 
> Cheers,
> Ralf

So long,
Igor


mail2web.com – Enhanced email for the mobile individual based on Microsoft®
Exchange - http://link.mail2web.com/Personal/EnhancedEmail




___
http://lists.gnu.org/mailman/listinfo/libtool

RE: Avoid la files.

[Duft Markus]

Good Morning ;)

Ralf Wildenhues  wrote:
> * Duft Markus wrote on Thu, Dec 13, 2007 at 11:27:34AM CET:
>> 
>> I'm not sure if i can easily put in HEAD, we use 1.5.24. Would it be
>> enough to get HEAD built somewhere, and bootstrap only the last
>> component with it (leave all libraries untouched)?
> 
> Yes.

Ok, i will try on the Linux box (when the box's owner is here).

> 
>> Maybe we can try on a linux box, w32 won't work, since i don't have
>> my patches ported for HEAD ;o(
> 
> Yes, that should be sufficient.  I can't tell you yet whether it would
> help, though, as I'm not yet sure about the actual problem cause.

On the other hand i don't know if this would lead to anything. Even if
it works with HEAD - i can't go away from 1.5 right now.

> 
>> I attached both the libtool call and the linker call from my w32
>> box, if you wanr i can organize linux output as well.
> 
> Thanks.
> 
> So I gather all of those libraries are installed shared libraries that
> come from a third-party package?  And many of those libraries
> themselves depend on other libraries?

Those are nearly all libraries that come from us. About 460 libraries
belong to the program directly, and about 300 libraries are part of out
so-called "toolsbox". Most of the libraries that belong to the program
have dependecies among each other, and all have dependencies to toolsbox
libraries (to many of them)

> 
>> There is not a single convenience library in play, since our tool
>> which handles these things (Confix,
>> http://www.sf.net/projects/confix) is still not able to produce
>> those things (arg I know... We'd really need it for our command
>> lines) 
> 
> I still haven't understood whether this is actually a problem in your
> setup/Config/your package, and you want libtool to work around those
> limitations, or you *really* need five hundred different shared
> libraries and a program that links against all of them.

Naa, libtool doesn't need to do anything different from what it is doing
now, it just needs to be faster ;o) Our build tools simply don't know
yet of how to tell libtool to build convenience libraries, but that will
be solved sometimes...

The number of libraries (without convenience libraries) is one of the
little things i really cannot influence ;o( This is the structure of our
software, i'll have to live with that...

> 
> I mean, wow, have you ever considered that the runtime linker overhead
> for this kind of setup can be excessive?  Just symbol searching is
> bound to take a looong time, and that's even on sane systems like
> GNU/Linux. 

Yes, i'm aware of that. BTW on windows this is really fast ;o)

> 
> Be kindly advised to consider some general guidance how to create
> shared (ELF) libraries:
> . 
> Note also that *fewer* shared libraries is a good thing for the
> reasons mentioned therein.

I know how to create shared libs, but this piece of software was never
designed to be shared. Or it was, but somebody missed it *lol*

> 
> However, if all those libraries should instead be convenience
> archives, because they should just be linked together into one (or a
> few) big shared libraries, then you can avoid the libtool performance
> problems by simply making those libraries convenience archives.

Yes, maybe, but that doesn't change the fact, that the algotithm for
reading .la files has big scalability problems. It would be really cool
if this would be fixed (or i get a patch for this issue), since my
quick-fix for now (commenting it out), also destroys usage of rpath,
since this information is contained in the .la files too.

I found a sed/grep combination, which i rewrote to try things out. This
didn't get me any performance boost. Also i tried linking a smaller
shared libraries with a few (10) objects and a few dependent libraries
(about 20 direct deps, and inderect about 100 deps...). This also has
more or less big performance problems. It takes about 1 minute with .la
reading, and 20 seconds without...

I don't think it should be too hard to reproduce... Just get yourseld
the library with the most dependencies on your system, and try once
normally, and the with lines 2159 through 2172 commented out in
ltmain.sh. This should give a great difference.

> 
>> The problem i think this command triggers is, that libtool looks at
>> some .la files most libraries depend on a lot of times. Maybe
>> caching la lookup results would be a solution, although i cannot
>> think of how to do this...
> 
> Hmm, then I may need to look at the full output.  But please don't
> send it just yet, please just answer the questions above.  Don't
> hesitate to ask if there is something you don't understand, but
> please be precise in your answers.

Whatever you want... ;o) Hope that is enough information.

Cheers, Markus

> 
> Thanks,
> Ralf


-- 
19. - 21. Februar 2008
Salomon Automation auf der LogiMAT 2008, Neue Messe Stuttgart, Deutschland
Halle 6, Stand 527