[sr #110901] libtool hangs indefinitely on windows when used in msys due to cmd.exe call bug

[Mitch]

Follow-up Comment #5, sr #110901 (group libtool):

My original quick-fix suggestion was to simply unset MSYS_ARG_CONV_EXCL in
libtool so the hack inplace would work.  

Later for my own GNU builds I started to just add a flag that controlled of
the cmd.exe call should be as it is "//c"  or without the MSYS hack, and only
a single slash.

Honestly though these are both poor as this function really leads to horrible
performance in windows.


Instead I would recommend something like:
https://github.com/mitchcapper/libtool/compare/add_bypass_path_conversion_opt

Yes it adds a new ENV var so nothing benefits from the fix initially.  In fact
due to the nature of the bug no project will likely implement the flag in
their build process (unless they used the forward slash notation with relative
which would ensure msys escaping would not happen.

Instead this would be a user level flag that they would set similar to how
they set MSYS2_ARG_CONV_EXCL to control escaping.

While it would potentially make more sense for it to be a libtool command line
arg the problem there is the user would likely need to edit the build
/configure scripts manually to do so which would be a big pain.

Now this straight up bypasses the conversion rather than using the singular
slash as I originally did with a flag. The reason being for properly
configured filenames no translation is needed and the performance hit with
even the correctly working cmd.exe call is pretty huge.  Windows has a "high"
cost for process spawning and that one command that is called on every path
spawns many processes.  Part of this is sh/bash.exe on windows spawns an
additional copy of itself every time it goes to launch a process.  In short a
single file conversion call causes:
A 'forked' sh.exe to then run'env.exe bash /usr/bin/cmd /c echo .libs/'.  That
spawns env.exe, which spawns bash.exe.   Thankfully echo is a builtin of
cmd.exe so no spawn there.   Then we use sed to trim the result so that is
another copy of sh.exe that then calls sed.exe.

All in all to convert ".libs/" to the proper path you are talking about 8
processes started and terminated in series.

So bypassing any spawns is a huge performance increase.

If we don't want to do this sort of bypass option we should still give users
some way to avoid the double slash hack without them resorting to editing
libtool or the build commands itself.  Heck most users probably just give up
instead when it freezes not knowing the issue.

One change we could do would be to be smarter about if we need to use the hack
at all.  Relative paths, for example, I don't think ever need to be converted.
 One could replace any backslashes with forward slashes as just about anything
windows should understand forward slashed paths.

Another option would be to potentially detect if cmd.exe needs a single slash
or double slash to be past in the configure script or just at the very start
of a libtool spawn.  That won't help performance but would prevent hangs.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/


signature.asc
Description: PGP signature

[sr #110901] libtool hangs indefinitely on windows when used in msys due to cmd.exe call bug

[Mitch]

Follow-up Comment #6, sr #110901 (group libtool):

Sorry, I forgot to add one way to slightly increase performance as this does
seem msys specific would be to call cygpath as we do elsewhere.  That would
reduce it from 8 to 2 processes spawned every time I believe.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/


signature.asc
Description: PGP signature

[sr #111157] MSVC / cl.exe .exp extension collision

[Mitch]

URL:
  

 Summary: MSVC / cl.exe .exp extension collision
   Group: GNU Libtool
   Submitter: mitchc
   Submitted: Mon 02 Dec 2024 08:47:48 PM UTC
Category: None
Priority: 5 - Normal
Severity: 3 - Normal
  Status: None
 Privacy: Public
 Assigned to: None
Originator Email:
 Open/Closed: Open
 Discussion Lock: Any
Operating System: Microsoft Windows


___

Follow-up Comments:


---
Date: Mon 02 Dec 2024 08:47:48 PM UTC By: Mitch 
I love libtool and it is great it is getting some additional Windows support.
Was happy to see the /Fe fix.  Unfortunately a biproduct of the newer MSVC
support has caused a new issue in terms of the .exp files using with
multi-step linking.

Libtool uses .exp files as input into cl.exe, and this works.  These are
essentially just command line commands in a file and using the "@" arg.  An
example is:
https://github.com/mitchcapper/libtool/blob/baa1fe41f9a2697e2d5847c8e1c97996560d56d1/m4/libtool.m4#L5742
.  The problem though is .exp is also used by microsoft for their own export
format:
https://learn.microsoft.com/en-us/cpp/build/reference/dot-exp-files-as-linker-input?view=msvc-140

So the first time through a make build it would succeed, when the .exp file is
used.  The command though will override the input file itself as microsoft
takes the output arg "-Fe$tool_output_objdir$soname"  and appends .exp for its
export file and this is the same name libtool uses.   This causes the make to
fail the second time with cl.exe complaining the file format is invalid (as it
is now a binary file not the additional command line args).

I use libtool with several GNU projects I build:
https://github.com/mitchcapper/WIN64LinuxBuild

and my libtool hacks:
https://github.com/mitchcapper/WIN64LinuxBuild/blob/0c7a19c857c8feac3e258669512f0abdb99d59f9/helpers_gnu.sh#L141-L142
just mass replace .exp with .expsym I have a commit of this specific fix at:
https://github.com/mitchcapper/libtool/compare/master...exp_extension_change

but I am not sure if this would be preferred for a patch.  As far as I am
aware .exp is only used internally by libtool so this should be decently
safe/not break anything.  Still you might prefer a windows only fix given MSVC
is the only place this happens.  Let me know and I can update the commit and
get it submitted.  

I read the process doc (
https://www.gnu.org/software/libtool/contribute.html#patches ) and understand
I should be making a changelog entry and submitting it to the mailing list
after posting it in this bug report.  I figured I would find out what
direction was desired before creating the patch.







___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/


signature.asc
Description: PGP signature

[sr #111158] 'make check' fails on older Mac, especially when building fat binaries

[anonymous]

URL:
  

 Summary: 'make check' fails on older Mac, especially when
building fat binaries
   Group: GNU Libtool
   Submitter: None
   Submitted: Tue 03 Dec 2024 07:15:19 AM UTC
Category: None
Priority: 5 - Normal
Severity: 3 - Normal
  Status: None
 Privacy: Public
 Assigned to: None
Originator Email: gstee...@gmail.com
 Open/Closed: Open
 Discussion Lock: Any
Operating System: Mac OS


___

Follow-up Comments:


---
Date: Tue 03 Dec 2024 07:15:19 AM UTC By: Anonymous
I'm using a Power Mac G5 with Mac OS 10.5.8 (Darwin 9.8).  My package manager
massages the build environment a bit to ensure consistent behaviour.  Part of
that involves ensuring that compiler runs always get "-arch " flags
inserted when a 64-bit or universal binary is being built (an absolute
requirement, because otherwise extension modules and test programs do not
necessarily get built with the same architecture as the rest of the package,
which causes immediate failure when (for example) a 64-bit program tries to
link to a 32-bit plug-in bundle, or the like).  In this environment, libtool's
test suite exhibits three incorrect failures and one incorrect skip.  I
believe that three of the four would still occur without it.

Test 95, for localization, artificially fails because it does not know to
ignore the additional error message from the 'lipo' tool that follows the
expected compiler error.

Test 119, with the enforced library prefix, fails because the linker refuses
to link an extension bundle (Mach filetype MH_BUNDLE) to a dynamic library
(Mach filetype MH_DYLIB).  I have yet to determine whether this is due to the
somewhat outdated native linker, or to a formal policy inherent to Mac OS.

Test 161, the Darwin fat compile, is incorrectly skipped.  As far as I can
tell this is gated by a very straightforward test for the string 'darwin'
appearing in the OS name (which it does, according to the logs), so how it is
deciding to skip that is a mystery to me.

Test 172, which repeats a large subset of the entire test suite with a shorter
maximum command line length, incorrectly reports failure when the only failed
tests are the same ones that failed at normal line lengths.  If nothing
changed, this should not be reporting a failure.







___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/


signature.asc
Description: PGP signature