Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Antoine Pitrou 
On Sat, 11 Oct 2014 00:30:51 + (UTC)
Sturla Molden  wrote:
> Larry Hastings  wrote:
> 
> > CPython doesn't require OpenBLAS.  Not that I am not receptive to the
> > needs of the numeric community... but, on the other hand, who in the
> > hell releases a library with Windows support that doesn't work with MSVC?!
> 
> It uses AT&T assembly syntax instead of Intel assembly syntax. 

But you can compile OpenBLAS with one compiler and then link it to
Python using another compiler, right? There is a single C ABI.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Sturla Molden 
Antoine Pitrou  wrote:

> But you can compile OpenBLAS with one compiler and then link it to
> Python using another compiler, right? There is a single C ABI.

BLAS and LAPACK are actually Fortran, which does not have a single C ABI.
The ABI depends on the Fortran compiler. g77 and gfortran will produce
different C ABIs. This is a consistent source of PITA  in any scientific
programming that combines C and Fortran.

There is cblas though, which is a C API, but it does not include LAPACK.

Another thing is that libraries are different. MSVC wants a .lib file, but
MinGW produces .a files like GCC does on Linux. Perhaps you can generate a
.lib file from a .a file, but I have never tried.

Sturla

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Antoine Pitrou 
On Sat, 11 Oct 2014 13:59:52 + (UTC)
Sturla Molden  wrote:
> Antoine Pitrou  wrote:
> 
> > But you can compile OpenBLAS with one compiler and then link it to
> > Python using another compiler, right? There is a single C ABI.
> 
> BLAS and LAPACK are actually Fortran, which does not have a single C ABI.
> The ABI depends on the Fortran compiler. g77 and gfortran will produce
> different C ABIs. This is a consistent source of PITA  in any scientific
> programming that combines C and Fortran.

But is that CPython's fault? I don't think so.

> Another thing is that libraries are different. MSVC wants a .lib file, but
> MinGW produces .a files like GCC does on Linux.

It sound like whatever MSVC produces should be the defacto standard
under Windows.

If Microsoft released a (GNU/)Linux compiler which produced
incompatible library files, I don't think anyone would regard them very
highly.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Sturla Molden 
Sturla Molden  wrote:

> BLAS and LAPACK are actually Fortran, which does not have a single C ABI.
> The ABI depends on the Fortran compiler. g77 and gfortran will produce
> different C ABIs. This is a consistent source of PITA  in any scientific
> programming that combines C and Fortran.
> 
> There is cblas though, which is a C API, but it does not include LAPACK.
> 
> Another thing is that libraries are different. MSVC wants a .lib file, but
> MinGW produces .a files like GCC does on Linux. Perhaps you can generate a
> .lib file from a .a file, but I have never tried.

And not to mention that the Fortran run-time depends on the C runtime...
What Carl Keffner did for SciPy was to use a static libgfortran, which is
not liked against any specific CRT, so it could be linked with msvcr90.dll
when the Python extension is built. The vanilla libgfortran.dll from MinGW
is linked with msvcrt.dll. However, not linking with msvcrt.dll broke the
pthreads library, which in turn broke OpenMP, so he had to patch the
pthreads library for this... This just shows some of the difficulties of
trying to combine the GNU and Microsoft compilers. There are many others,
like different stack alignment, differenr exception handling, and the mingw
runtime (which causes segfaults when linked dynamically to MSVC
executables). It's not just getting the CRT right.

Sturla

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Sturla Molden 
Antoine Pitrou  wrote:

> It sound like whatever MSVC produces should be the defacto standard
> under Windows.

Yes, and that is what Clang does on Windows. It is not as usable as MinGW
yet, but soon it will be. Clang also suffers fronthe lack of a Fortran
compiler, though.

Sturla

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Steve Dower 
Is there some reason the Fortran part can't be separated out into a DLL? That's 
the C ABI Antoine was referring to, and most compilers can generate import 
libraries from binaries, even if the original compiler produced then in a 
different format.

Top-posted from my Windows Phone

From: Sturla Molden
Sent: ‎10/‎11/‎2014 7:22
To: [email protected]
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

Antoine Pitrou  wrote:

> It sound like whatever MSVC produces should be the defacto standard
> under Windows.

Yes, and that is what Clang does on Windows. It is not as usable as MinGW
yet, but soon it will be. Clang also suffers fronthe lack of a Fortran
compiler, though.

Sturla

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Nathaniel Smith 
On 11 Oct 2014 14:42, "Antoine Pitrou"  wrote:
>
> On Sat, 11 Oct 2014 00:30:51 + (UTC)
> Sturla Molden  wrote:
> > Larry Hastings  wrote:
> >
> > > CPython doesn't require OpenBLAS.  Not that I am not receptive to the
> > > needs of the numeric community... but, on the other hand, who in the
> > > hell releases a library with Windows support that doesn't work with
MSVC?!
> >
> > It uses AT&T assembly syntax instead of Intel assembly syntax.
>
> But you can compile OpenBLAS with one compiler and then link it to
> Python using another compiler, right? There is a single C ABI.

In theory, yes, but this is pretty misleading. The theory has been known
for years. In practice we've only managed to pull this off for the first
time within the last few months, and it requires one specific build of one
specific mingw fork with one specific set of build options, and only one
person understands the details. Hopefully all those things will continue to
be available and there aren't any showstopper bugs we haven't noticed yet...

(Before that we've spent the last 5+ years using a carefully preserved
build of an ancient 32 bit mingw and substandard BLAS .dll that were handed
down from our ancestors; we've had no capability to produce 64 bit official
builds at all. And a large proportion of users have been using 3rd party
proprietary builds.)

-n
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Sturla Molden 
Steve Dower  wrote:

> Is there some reason the Fortran part can't be separated out into a DLL? 

DLL hell, I assume. Using the Python extension module loader makes it less
of a problem. If we stick with .pyd files where everything is statically
linked we can rely on the Python dev team to make sure that DLL hell does
not bite us. Most of the contributors to projects like NumPy and SciPy are
not computer scientists. So the KISS principle is important, which is why
scientific programmers often use Fortran in the first place. Making sure
DLLs are resolved and loaded correctly, or using stuff like COM or .NET to
mitigate DLL hell, is just in a different league. That is for computer
engineers to take care of, but we are trained as physicists, matematicians,
astronomers, chemists, biologists, or what ever... I am sure that engineers
at Microsoft could do this correctly, but we are not the kind of guys you
would hire :-)


OT: Contrary to common belief, there is no speed advantage of using Fortran
on a modern CPU, because the long pipeline and the hierarchical memory
alleviates the problems with pointer aliasing. C code tends to run faster
then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster
than C. In 2014, Fortran is only used because it is easier to program for
non-specialists. And besides, correctness is far more important than speed,
which is why we prefer Python or MATLAB in the first place. If you ever see
the argument that Fortran is used because of pointer aliasing, please feel
free to ignore it.


Sturla

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Steve Dower 
DLLs linked by import library at compile time (ie. not using LoadLibrary calls) 
and placed in the same directory as the .pyd should be exempt from DLL hell - 
Python already creates an activation context when importing pyds to let them 
load their own dependencies. Multiple CRTs are also okay as long as they don't 
try and share state (such as file descriptors) across boundaries.

I do understand the lack of knowledge and experience though. I've helped out in 
the past but I personally don't have the bandwidth to contribute more, nor the 
persuasiveness to get someone else to.

Top-posted from my Windows Phone

From: Sturla Molden
Sent: ‎10/‎11/‎2014 9:59
To: [email protected]
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

Steve Dower  wrote:

> Is there some reason the Fortran part can't be separated out into a DLL?

DLL hell, I assume. Using the Python extension module loader makes it less
of a problem. If we stick with .pyd files where everything is statically
linked we can rely on the Python dev team to make sure that DLL hell does
not bite us. Most of the contributors to projects like NumPy and SciPy are
not computer scientists. So the KISS principle is important, which is why
scientific programmers often use Fortran in the first place. Making sure
DLLs are resolved and loaded correctly, or using stuff like COM or .NET to
mitigate DLL hell, is just in a different league. That is for computer
engineers to take care of, but we are trained as physicists, matematicians,
astronomers, chemists, biologists, or what ever... I am sure that engineers
at Microsoft could do this correctly, but we are not the kind of guys you
would hire :-)


OT: Contrary to common belief, there is no speed advantage of using Fortran
on a modern CPU, because the long pipeline and the hierarchical memory
alleviates the problems with pointer aliasing. C code tends to run faster
then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster
than C. In 2014, Fortran is only used because it is easier to program for
non-specialists. And besides, correctness is far more important than speed,
which is why we prefer Python or MATLAB in the first place. If you ever see
the argument that Fortran is used because of pointer aliasing, please feel
free to ignore it.


Sturla

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Nathaniel Smith 
I'm not at all an expert on Fortran ABIs, but I think there are two
distinct issues being conflated here.

The first is that there is no standard way to look at some Fortran source
code and figure out the corresponding C API. When trying to call a Fortran
routine from C, then different Fortran compilers require different sorts of
name mangling, different ways of mapping Fortran concepts like "output
arguments" onto C concepts like "pointers", etc., so your C code needs to
have explicit knowledge of which Fortran compiler is in use. That's what
Sturla was referring to with the differences between g77 versus gfortran
etc. This is all very annoying, but it isn't a *deep* bad - it can be
solved by unilateral action by whatever project wants to link the Fortran
library.

The bigger problem is that getting a usable DLL at all is a serious
challenge. Some of the issues we deal with: (a) the classic, stable mingw
has no 64-bit support, (b) the only portable way to compile fortran (f2c)
only works for the ancient fortran 77, (c) getting even mingw-w64 to use a
msvc-compatible ABI is not trivial (I have no idea why this is the case,
but it is), (d)
mingw-built
dlls normally depend on the mingw runtime dlls. Because these aren't
shipped globally with Python, they have to be either linked statically or
else a separate copy of them has to be placed into every directory that
contains any mingw-compiled extension module.

All the runtime and ABI issues do mean that it would be much easier to use
mingw(-w64) to build extension modules if Python itself were built with
mingw(-w64). Obviously this would in turn make it harder to build
extensions with MSVC, though, which would be a huge transition. I don't
know whether gcc's advantages (support for more modern C, better
cross-platform compatibility, better accessibility to non-windows-experts,
etc.) would outweigh the transition and other costs.

As an intermediate step, there are almost certainly things that could be
done to make it easier to use mingw-w64 to build python extensions, e.g.
teaching setuptools about how to handle the ABI issues. Maybe it would even
be possible to ship the mingw runtimes in some globally available location.

-n
On 11 Oct 2014 17:07, "Steve Dower"  wrote:

>   Is there some reason the Fortran part can't be separated out into a
> DLL? That's the C ABI Antoine was referring to, and most compilers can
> generate import libraries from binaries, even if the original compiler
> produced then in a different format.
>
> Top-posted from my Windows Phone
>  --
> From: Sturla Molden 
> Sent: ‎10/‎11/‎2014 7:22
> To: [email protected]
> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows
>
>   Antoine Pitrou  wrote:
>
> > It sound like whatever MSVC produces should be the defacto standard
> > under Windows.
>
> Yes, and that is what Clang does on Windows. It is not as usable as MinGW
> yet, but soon it will be. Clang also suffers fronthe lack of a Fortran
> compiler, though.
>
> Sturla
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Paul Moore 
On 11 October 2014 19:32, Nathaniel Smith  wrote:
> The bigger problem is that getting a usable DLL at all is a serious
> challenge. Some of the issues we deal with: (a) the classic, stable mingw
> has no 64-bit support, (b) the only portable way to compile fortran (f2c)
> only works for the ancient fortran 77, (c) getting even mingw-w64 to use a
> msvc-compatible ABI is not trivial (I have no idea why this is the case, but
> it is), (d) mingw-built dlls normally depend on the mingw runtime dlls.
> Because these aren't shipped globally with Python, they have to be either
> linked statically or else a separate copy of them has to be placed into
> every directory that contains any mingw-compiled extension module.

These are all genuine and difficult issues. I have some knowledge of
the mingw environment, although only as a user, and I can confirm that
it is not in an ideal state. As you mention, the core project only
offers 32-bit compilers, and the 64-bit versions are separate, not
always reliable, projects. It can be the case that getting a
successful build relies on using a precise distributed archive of the
relevant mingw binaries.

By default, mingw uses msvcrt, for essentially licensing reasons (to
do with the fact that msvcrt is the only version they can consider as
being "shipped with the OS" which is relevant to their use of the
GPL). To get it to link to an alternative VC runtime requires a symbol
library for that runtime, which needs someone to build it. I don't
know whether mingw includes runtime symbol libraries for later than
msvcr10[1] - I asked for that to be added when Python switched to
VC10, and it was fairly difficult to get that done, even at that point
when the mingw community was much less fragmented than now.

It is possible to build C extensions with mingw in such a way that
they don't depend on the mingw runtime DLLs - at least the distutils
support made that happen for the average extension when I last worked
on it which was pre-Python 3... I'm pretty sure that C++ needs runtime
support DLLs which would be tricky to avoid, and I have no idea what
sorts of difficulty Fortran might add (although your comments sound
about what I expect).

> All the runtime and ABI issues do mean that it would be much easier to use
> mingw(-w64) to build extension modules if Python itself were built with
> mingw(-w64). Obviously this would in turn make it harder to build extensions
> with MSVC, though, which would be a huge transition. I don't know whether
> gcc's advantages (support for more modern C, better cross-platform
> compatibility, better accessibility to non-windows-experts, etc.) would
> outweigh the transition and other costs.

As mentioned above, I don't think the mingw environment is that
reliable, which would be an issue if Python were built with it. Would
it really be a positive step if we had to say "to build Python you
need to download this precise personal build from a specific mingw64
spin-off project"? And yes, I have code for which that is *precisely*
what I need to do.

> As an intermediate step, there are almost certainly things that could be
> done to make it easier to use mingw-w64 to build python extensions, e.g.
> teaching setuptools about how to handle the ABI issues. Maybe it would even
> be possible to ship the mingw runtimes in some globally available location.

As I've said a number of times now, I think this is much more likely
to be a useful avenue. For example, shipping the appropriate
libmsvcrXXX.a and static libraries for the relevant runtimes with
distutils, instead of relying on the user having a version of mingw
that does so. And testing (and fixing if necessary) the distutils
MingwCompiler class with a wider range of mingw builds. Note that
where I say distutils here, it would ideally be something that we do
in setuptools, so that it won't be tied to the stdlib release cycles.
But AFAIK, setuptools doesn't yet include any compiler classes, so
that'd be a bigger change. I have no idea what the most appropriate
direction to take would be here.

By the way, Steve Dower - distutils\cygwinccompiler will need the new
MSVC runtime added to get_msvcr() for 3.5/VC15. It won't help
unless/until mingw ships the runtime symbol library, of course, but if
it's not added to the shipped Python 3.5, it'll be a pain to add
later... It doesn't seem to be in your VC14 branch.

Paul

[1] I just checked and it seems that there's a msvcr110 library
shipped with the mingw I have, but nothing later.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-11 Thread Nick Coghlan 
On 12 Oct 2014 04:20, "Steve Dower"  wrote:
>
> DLLs linked by import library at compile time (ie. not using LoadLibrary
calls) and placed in the same directory as the .pyd should be exempt from
DLL hell - Python already creates an activation context when importing pyds
to let them load their own dependencies. Multiple CRTs are also okay as
long as they don't try and share state (such as file descriptors) across
boundaries.
>
> I do understand the lack of knowledge and experience though. I've helped
out in the past but I personally don't have the bandwidth to contribute
more, nor the persuasiveness to get someone else to.

The current key phrase in getting potential corporate sponsors interested
in the scientific Python stack is "big data analytics". The professional
programming world is actually quite bad at using modern mathematical
techniques effectively - the scientific community are way ahead of us.
Python is relatively unique in being a language used extensively by both
professional programmers *and* research scientists, so we're well
positioned to serve as a basis for effective communication between the two
groups.

AMPLab at Berkeley are one of the groups at the forefront of that. MS are
sponsors, so if you can find the group behind that sponsorship, you may
find folks in a position to help out with the scientific Python toolchain
challenges.

Cheers,
Nick.

>
>
> Top-posted from my Windows Phone
> 
> From: Sturla Molden
> Sent: ‎10/‎11/‎2014 9:59
>
> To: [email protected]
> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows
>
> Steve Dower  wrote:
>
> > Is there some reason the Fortran part can't be separated out into a
DLL?
>
> DLL hell, I assume. Using the Python extension module loader makes it less
> of a problem. If we stick with .pyd files where everything is statically
> linked we can rely on the Python dev team to make sure that DLL hell does
> not bite us. Most of the contributors to projects like NumPy and SciPy are
> not computer scientists. So the KISS principle is important, which is why
> scientific programmers often use Fortran in the first place. Making sure
> DLLs are resolved and loaded correctly, or using stuff like COM or .NET to
> mitigate DLL hell, is just in a different league. That is for computer
> engineers to take care of, but we are trained as physicists,
matematicians,
> astronomers, chemists, biologists, or what ever... I am sure that
engineers
> at Microsoft could do this correctly, but we are not the kind of guys you
> would hire :-)
>
>
> OT: Contrary to common belief, there is no speed advantage of using
Fortran
> on a modern CPU, because the long pipeline and the hierarchical memory
> alleviates the problems with pointer aliasing. C code tends to run faster
> then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster
> than C. In 2014, Fortran is only used because it is easier to program for
> non-specialists. And besides, correctness is far more important than
speed,
> which is why we prefer Python or MATLAB in the first place. If you ever
see
> the argument that Fortran is used because of pointer aliasing, please feel
> free to ignore it.
>
>
> Sturla
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com