[Back on list, just to make sure it's archived. Max and I have bounced
some emails back-n-forth privately, so you're coming in on the middle of
a conversation... See Gen Zhang's original message on 12/4/2002]
Also, this is kinda stream of conciousness...because I end up at the end
of the message somewhere other than where I thought I was going... <g>
Okay, from Zhang's report:
> After re-libtoolize-ing the sources (to allow it to produce shared
> libs) the compilation is fine through glib and will produce
> libglib-2.0.la, but will complain about undefined references to
> things like '_g_free' and '_g_log' (which I presume is in
> libglib-2.0.la). I've attached the output from make (upon a second
> run, after it going through glib).
The error was not in creating the DLL in the first place(*). The error was
> /bin/bash ../libtool --mode=link gcc -g -O2 -Wall -D_REENTRANT -o
> glib-genmarshal.exe glib-genmarshal.o ../glib/libglib-2.0.la -liconv
> -lintl -liconv
> gcc -g -O2 -Wall -D_REENTRANT -o .libs/glib-genmarshal.exe
> glib-genmarshal.o ../glib/.libs/libglib-2.0.dll.a -L/usr/lib
> -L/lib/w32api -luser32 -lkernel32 /usr/lib/libintl.dll.a
> /usr/lib/libiconv.dll.a -Wl,--rpath -Wl,/usr/local/lib
> glib-genmarshal.o(.text+0x1807): In function `complete_in_arg':
> /usr/src/glib-2.0.7/gobject/glib-genmarshal.c:218: undefined reference
> to `_g_log'
> ...
(*) unless the DLL itself exported no symbols at all. But, new libtool
DOES NOT USE def files anymore. AT ALL. (and, you can't even give it
one as an argument...)
Max Bowsher wrote:
>>> .def files are passed as -Wl,-retain-symbols-file deffile, not,
>>> as is correct, the first 'object' file.
Which means this ^^^^^ is wrong.
"-Wl,--retain-symbols-file -Wl,filename" is perfectly okay -- see the ld
manpage. However, $export_symbols contains a filename whose contents
are a list of exports -- like a def file, but not exactly; it's more
unix-focused. If you trace the code in ld.exe, you find that
--retain-symbol-file's argument is handled in a COMPLETELY different
manner than a def file would be (it mainly acts to prevent the given
symbols from being stripped; it doesn't "trigger" the necessary hooks to
export those symbols in a DLL on cygwin/mingw). So,
--retain-symmbols-file is okay on other unices, but not really the right
switch for libtool to pass to ld, on cygwin.
On cygwin, what you end up with is
"don't strip these symbols"
and
"there is no def file, so by default, export all symbols(*)"
(*) unless there are symbols that have been specifically decorated with
__declspec(dllexport), in which case ONLY those decorated symbols are
exported. It turns out that THIS is the root of the problem with
glib+cygwin.
So, on cygwin, this definition of archive_expsym_cmds is probably wrong
-- but not in the way you thought. Because **archive_expsym_cmds** is
NOT USED (much). [except in glib...sigh...]
Cygwin defines "always_export_symbols" as "no" (unless you use the
'--export-symbols <FILENAME>' switch when calling libtool), and
"export_symbols_regex" is empty unless you explicitly call libtool with
the '--export-symbols-regex <REGEX>' switch. Given those typical
definitions, we use "archive_cmds" NOT "archive_expsym_cmds" almost 100%
of the time. Only when either '--export-symbols <FILENAME>' or
'--export-symbols-regex <REGEX>' are specified do we, on cygwin, use
"archive_expsym_cmds".
And glib does. It uses '--export-symbols glib.def' (on OS_WIN32, but
not CYGWIN). More below.
Further, as far as ld is concerned, the def file doesn't need to be the
FIRST object file. That was the case for dlltool (or you could
explicitly say "--def def.file"), but not for ld once the functionality
was absorbed. For ld, the def file just needs to appear as one of the
arguments to ld. (see gld_${EMULATION_NAME}_unrecognized_file in
binutis/ld/emultempl/pe.em)
> But I'm worried about just assuming that the -export-symbols file is
> a def file. There was some code in libtool somewhere (recently
> removed) which searched for EXPORTS to try and determine if it was a
> def file, and else tries to make one.
>
> I'm not sure how export-symbols works on other platforms.
Explained above. Now, IF someone were to specify --export-symbols-regex
or --export-symbols on cygwin, then the archive_expsyms_cmd would be
used -- and we need something like
_LT_AC_TAGVAR(archive_expsym_cmds, $1)='$CC -shared -nostdlib
$predep_objects $libobjs $deplibs $postdep_objects $compiler_flags -o
$output_objdir/$soname $export_symbols
${wl}--out-implib,$lib'
(e.g. no retain-symbols argument, just include $export_symbols as one of
the "object" files)
There's just one problem: ld tests "unknown objects" filenames to see if
they end in ".def" or ".DEF". $export_symbols is (usually) libname.exp.
Badness.
So, for this to work as we expect, we must
(a) change $export_symbols to "libname.def" not "libname.exp". Old
versions of libtool (back in dlltool days) did
cp $export_symbols $output_objdir/$soname-def
(dlltool was less picky about the actual name of the .def file; you'd
just say "--def foo" and it was happy. We'd need to
cp $export_symbols $output_objdir/$soname.def
to insure that the file ended with .def.
(b) change $export_symbols_cmds so that it prepends "EXPORTS". Now,
since always_export_symbols is false by default, the only time
$export_symbols_cmds ever gets called is when you need to GENERATE an
exports list -- and since '--export-symbols <FILENAME>' takes an
argument, you don't need to generate it in that case (it's in
<FILENAME>). So, the only time, on cygwin, that $export_symbols_cmds is
ever called, is when you've done '--export-symbols-regex <REGEX>' Which
makes things EVER so much more complicated. Sigh.
Anyway, perhaps something like $export_symbols_cmds =
"\$NM \$libobjs \$convenience | \$global_symbol_pipe | \$SED 's/.* //'
| sort | uniq | \$SED -e '1iEXPORTS' > \$export_symbols"
(note the final $SED command)
However, that still doesn't differentiate between DATA and function
exports; the .def file, if used, must identify DATA exports. Which
means $export_symbols_cmds has be a LOT more complicated:
$NM $libobjs $convenience | \$global_symbol_pipe | \
\$SED -e '/^[BCDG] /s/^\([BCDG]\) \(_[_A-Za-z0-9]*\)
\([_A-Za-z0-9]*\)/\3 DATA/' | \
\$SED -e '/^[AISTW] /s/^\([AISTW]\) \(_[_A-Za-z0-9]*\)
\([_A-Za-z0-9]*\)/\3/' | \
sort | uniq | \$SED -e '1iEXPORTS' > \$export_symbols"
Old libtools used to use a specialized program (impgen) to do this, and
a combination of dlltool calls. But we want to stay away from that, and
stick with ld, gcc, and nm. So, we'll have to work harder...
Worse, if you actually specify --export-symbols=FOO on the libtool
command line (e.g "use the file 'FOO', which already exists and
specifies the symbols to export, as $export_symbols") then
$export_symbols_cmds is NOT called, AND you lose platform
cross-compatibility in the $export_symbols file. On unix, the file
$export_symbols should just contain
bob
fred
But on windows, it should contain
EXPORTS
bob
fred DATA
because we're using it (in my "proposal" above) as a def file.
Basically, what you're trying to do is build an .def file on the fly.
This USED to be very helpful -- before the auto-import, auto-export
functionality was added to ld.exe itself. Now, it just messes things up
on cygwin. (It might still be useful on other platforms, but not here).
I think this whole concept (on-the-fly def file created by libtool) was
a kludge, because dlltool NEEDED a def file (during its 'make a dll'
mode), so libtool created one (by calling dlltool in its 'make a def
file' mode). Now, we don't use dlltool anymore, and ld can
automatically figure out what, and how (DATA or function) to export. So
none of this is needed on windows anymore, in general.
The only time you need any of this "specially restrict the export list
or specifically export a known list of symbols" is when you're doing
something *special*. And that requires developer intervention....
In cygwin/mingw, the typical library needs to do one of the following:
1) export everything (almost always, this is what you want)
2) export everything EXCEPT ...
3) export only ...
4) export everything, BUT you've specifically decorated some symbols
(like structs) with __declspec(dllexport). If it weren't for libtool's
paranoia about passing flags to the linker, you'd just say
'--export-all' and be done. But libtool won't let you do that, so life
is hard. This is the glib case.
If you want #1, you just let ld do the work. No need for export_symbols
or any of that junk.
If you want #2, the *Developer* should specify that in the makefile.
E.g. libfoo_la_LDFLAGS=.... --exclude-symbols SYMBOL,SYMBOL,... or
--exclude-libs LIB,LIB,... (to avoid exporting symbols pulled in from
static convenience libraries). But that's a *developer* thing; libtool
shouldn't attempt to do that itself. It isn't clairvoyant. Perhaps the
developer should insure that the Makefile only puts --exclude-symbols
... when building for cygwin/mingw by using conditionals. But it's a
developer thing.
Now, if on unix, a developer used --export-symbols or
--export-symbols-regex as a means of NOT exposing certain symbols to
clients, then we need to take special care when porting to windows.
Create a .def file and add it (conditionally) to the _SOURCES [BUZZ!
doesn't work], or add --exclude-[lib|symbol] to _LDFLAGS (conditionally)
[BUZZ! Doesn't work. Damn, but libtool is picky about the flags it
passes to the linker], etc. But this should be RARE.
If you want #3, then the *developer* should specify a .def file as one
of the library's dependencies (perhaps conditional on cygwin/mingw). End
of story.
And --export-symbols/--expoprt-symbols-regex should continue to be
handled on cygwin exactly as it is handled on other unixes: strip or
don't strip. It shouldn't affect the "windowsish" exporting process.
That's the job of --exclude-[lib|symbobs] and .def file, if such
functionality is needed.
But apparently Tor Lilqvist disagrees:
Now, having said all of that, it appears that this is EXACTLY what the
glib folks have done. They explicitly add "-export-symbols glib.def" IF
and ONLY IF OS_WIN32 (eg. not for cygwin, which has PLATFORM_WIN32 true,
but OS_WIN32 false). IMO, that is a *misunderstanding* of what
--export-symbols does in CURRENT libtool, even if it was correct for OLD
libtool -- but Tor Lilqvist [the gtk on windows dude] uses a forked
version of libtool anyway, so...
As it happens, they've managed to outsmart the auto-export code when
they try this OLD libtool usage with a NEW libtool program. Auto-export
ONLY happens if BOTH of the following are true
1) there is no .def file
2) there are no explicit __declspec(dllexport) decorations on any
symbols.
1) is true, because the .def file is only used for OS_WIN32 (not
cygwin). However, 2) is false, because there are five symbols that are
explicitly decorated:
g_io_watch_funcs in glib/giowin32.c
g_timeout_funcs in glib/gmain.c
g_idle_funcs in glib/gmain.c
g_ascii_table in glib/gstrfuncs.c
g_thread_functions_for_glib_use in glib/gthread.c
These are the only DATA exports from glib that are "complex" (see 'info
ld' and search on 'auto-import').
So, the "right thing" on cygwin is one of these three things:
include the .def file -- but as explained exhaustively above, you can't
do it the way Tor does, the (new) "real" libtool doesn't use
--export-symbols the way he thinks (although the old libtool did). So,
you'd have to conditionally include it as a part of glib_la_SOURCES. But
THAT doesn't work EITHER -- because libtool says "what's this random
file doing in my argument list, it's not a .lo, it's not .la, throw it
out" So, Tor's solution is the only one that can work -- but it
requires that $archive_expsyms_cmd be fixed up etc etc... (*)
or
add --export-all-symbols as one of the glib_la_LDFLAGS switches (this
will re-enable the auto-export feature of ld, even though SOME symbols
are explicitly decorated). However, libtool does not pass
"--export-all-symbols" thru to the linker, so this might require a patch
to libtool. Probably not the best way to do this.
or
remove the explicit decorations on those five variables, and rely on the
runtime-pseudo-relocs feature of binutils. This won't work until
cygwin-1.3.18 is released; it's a very new feature. And
--enable-runtime-pseudo-relocs is not the default for ld yet, so you
need to add that flag to LDFLAGS...but libtool refuses to pass that
switch to the linker! So this won't work until (a) the new cygwin
kernel comes out, and (b) --enable-runtime-pseudo-relocs becomes the
default on cygwin's ld. (b) won't happen for quite some time, yet.
-----
So, it appears that that the only way to address this in the short term
is...restore some form of the old libtool-1.4.x code for
$archive_expsym_cmd.
Fixup the official libtool so that IF '--export-symbols <FILENAME>' is
specified, then on cygwin/mingw assume that the file contains a
.def-format specification, and $archive_expsyms_cmd includes it as "just
one of the object files" and doesn't use the --retain-symbols-file at
all. (On other platforms, go ahead and use --retain-symbols-file, etc).
This means that it would be up to the developer to do the following,
IF he wants to restrict the list of export symbols on both unix and windows:
1) provide foo.exp which is just a list of symbols
2) provide foo.def which is a def-format list of symbols, with DATA
specifiers, etc
3) conditionally add "--export-symbols foo.exp" on unix, and
"--export-symbols foo.def" on windows.
This would require all of the changes to $export_symbols_cmds described
above, too, so that libtool's --export-symbols-regex switch could still
work (currently, that is hopelessly broken on cygwin).
Now that I've thought about it, I believe that this IS what should
happen in the libtool codebase. It seems that some of the "old stuff"
that was kept around for dlltool and was recently removed, should be
retained so that the new-and-improved,
use-auto-import-as-much-as-possible libtool can properly handle corner
cases like these.
But, that's a separate issue from "how to get glib working on cygwin".
IF all of those changes were in the current libtool, then yeah -- do it
that way. But they are not (yet). The long-term solution is to remove
the __declspec decorators on those five symbols when __CYGWIN__, and
rely on --enable-runtime-pseudo-relocs once it becomes default, and on
the latest cygwin kernel which should be out in a few days. (If you
wanted to try this method out BEFORE --enable-runtime-pseudo-relocs
becomes the default, the only way to pass that flag thru libtool appears
to be create a shell script and pretend that it is your compiler:
--gcc--
/path/to/real/gcc -Wl,--enable-runtime-pseudo-relocs $*
-------
Because libtool is REALLY picky about what it passes thru to the linker.
Sheesh!
Eventually pseudo-relocs will be the default on cygwin -- and maybe
mingw -- but not for a number of months, I'm sure.)
Anyway, thanks for bringing up this issue; it appears that the current
libtool is not quite ready for absolutely every possible use that people
want...Looks like there are a few things I need to do over Christmas
break... :-)
--Chuck
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
