Re: [Python-Dev] PEP 475

[Antoine Pitrou]

On Wed, 29 Oct 2014 00:05:57 +0100
Victor Stinner  wrote:
> 2014-10-28 22:36 GMT+01:00 Antoine Pitrou :
> > Is there an implementation somewhere?
> 
> There is no implementation yet. This time, I chose to focus on the PEP
> before working on an implementation :-)
> 
> We can work on the implementation if it helps discuss the PEP. I
> created a repository 3 months ago, but it has no commit yet:
> https://hg.python.org/features/eintr/

An implementation would help assess whether the change breaks existing
code or not (in the stdlib; I am not asking you to test third-party
packages :-)).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

"You might want to interact with Steve Dower off-list"

FWIW, I'm happy to talk specifics off list, and have already been involved in a 
number of discussions with the numpy and Scipy guys wrt figuring out specific 
technical challenges or clarifying non obvious parts of dealing with Windows. 
(As far as coding goes, practically all my spare time is taken up already, 
which is why I haven't contributed directly to those projects, much as I'd like 
to.)

Cheers,
Steve

Top-posted from my Windows Phone

From: Stephen J. Turnbull
Sent: ‎10/‎28/‎2014 20:59
To: Tony Kelman
Cc: python-dev@python.org
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

Tony Kelman writes:

 > No, just hearing the words come out of my mouth they sound a little
 > nuts.  Maybe not, there are after all half a dozen or more
 > incompatible alternate Python implementations floating around. I
 > think most of them started as from-scratch rewrites rather than
 > source forks, but maybe that's irrelevant.

Well, they have different names and they clearly are not intended to
be ABI compatible, so noone expects that.  OTOH, there clearly is an
expectation among many (and not just in the Windows world, cf. all of
the distros that provide whole stacks of everything for each version
of Python) that downloaded packages will just work without
incompatibility.

 > > Well, for starters, most of python-dev would rather avoid any contact
 > > whatsoever with Windows.  I think part of the problem is that Windows
 > > developers *of* Python are *very* rare (fingers of one hand rare).
 >
 > In my opinion the MSVC toolchain makes that problem worse, as it's far
 > harder for unix developers to have any familiarity with how things
 > work.

I've used Cygwin, I've used MinGW, and I've used VC.  Sure, the former
two are GCC-based so I have a lot of muscle memory for command-line
switches.  But that's not very important; the pain of using Windows is
what drives me away from all of them.

 > But you do have involvement and core developers from Microsoft
 > which is obviously incredibly important. Maybe even mandatory for
 > Python on Windows to be viable in your eyes.

No, I don't think that's true.  What I think *is* true is that most
developers on Windows do have access to Microsoft tools, so we do need
to provide compatibility with them.  As you say, the VC toolchain is
not all things to all men, but what's visible to python-dev makes it
more important than Cygwin or MinGW.  See Paul Moore's post about
communications between the scientific Python community and python-dev
for what I mean by "visible".

 > > It should be evident by now that our belief is that the large
 > > majority of Windows users is well-served by the current model
 >
 > This is not the case at all in the scientific community. NumPy and
 > SciPy put in a lot of extra work to come up with something that is
 > compatible with the MSVC build of CPython because they have to, not
 > because they're "happy to" jump through the necessary hoops.

Agreed.  This is well-known to python-dev, and AFAICS it *is* a
concern for us.  However, as Paul points out, a bridge needs to be
built.  Your posts have been a contribution to building that bridge,
for sure, but more work on the bridge is needed.

 > Do python-dev and numpy-discussion not talk to one another?

Exactly the issue here.  To resolve this, we need to talk more.
Unfortunately, I'm not one to help build the bridge as I haven't
developed on Windows at all since about 2003.

 > I'm going to move the "extensions with MinGW-w64" part of this
 > conversation over to numpy-discussion,

As far as I can tell, that's a good idea right now.  They have the
need, they have the expertise, both of which are somewhat lacking
here.

 > Okay. I'll table the discussion with python-dev for now then.

I hope you'll be able to come pick it back up at some point.  You
might want to interact with Steve Dower off-list, as he's spearheading
the effort to move the official builds to the "stable ABI" version of
MSVC.  Once that's in place, the MinGW guys will have a stationary
target which is up to date to shoot at.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:

> most developers on Windows do have access to Microsoft tool

I assume you mean python-dev folks who work on Windows:  it is certainly
not true for the vast majority of develoeprs who use Python on Windows,
who don't have the toolchain build their own C extensions.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlRQ+BYACgkQ+gerLs4ltQ7L8QCeJ4NYs+//39O0dUUNqG1lXy1Z
7GMAniUCjmodCfKAVBF/yeOv3GrR03Fm
=n4bm
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver  wrote:
> On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
> 
> > most developers on Windows do have access to Microsoft tool
> 
> I assume you mean python-dev folks who work on Windows:  it is certainly
> not true for the vast majority of develoeprs who use Python on Windows,
> who don't have the toolchain build their own C extensions.

I'm pretty sure he meant "most people who develop software for Windows",
even though that's not how he phrased it.  But this does not include, as
you point out, people who develop Python software that *also* works on
Windows.

If you are writing code targeted for Windows, I think you are very
likely to have an MSDN subscription of some sort if your package includes C
code.  I'm sure it's not 100%, though.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Wed, 29 Oct 2014 10:31:50 -0400
"R. David Murray"  wrote:

> On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver  wrote:
> > On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
> > 
> > > most developers on Windows do have access to Microsoft tool
> > 
> > I assume you mean python-dev folks who work on Windows:  it is certainly
> > not true for the vast majority of develoeprs who use Python on Windows,
> > who don't have the toolchain build their own C extensions.
> 
> I'm pretty sure he meant "most people who develop software for Windows",
> even though that's not how he phrased it.  But this does not include, as
> you point out, people who develop Python software that *also* works on
> Windows.
> 
> If you are writing code targeted for Windows, I think you are very
> likely to have an MSDN subscription of some sort if your package includes C
> code.  I'm sure it's not 100%, though.

You can use Express editions of Visual Studio.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Nick Coghlan]

On 30 October 2014 00:46, Antoine Pitrou  wrote:
> On Wed, 29 Oct 2014 10:31:50 -0400
> "R. David Murray"  wrote:
>
>> On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver  
>> wrote:
>> > On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
>> >
>> > > most developers on Windows do have access to Microsoft tool
>> >
>> > I assume you mean python-dev folks who work on Windows:  it is certainly
>> > not true for the vast majority of develoeprs who use Python on Windows,
>> > who don't have the toolchain build their own C extensions.
>>
>> I'm pretty sure he meant "most people who develop software for Windows",
>> even though that's not how he phrased it.  But this does not include, as
>> you point out, people who develop Python software that *also* works on
>> Windows.
>>
>> If you are writing code targeted for Windows, I think you are very
>> likely to have an MSDN subscription of some sort if your package includes C
>> code.  I'm sure it's not 100%, though.
>
> You can use Express editions of Visual Studio.

And the PSF can help out if folks working on Python software for
Windows need an MSDN subscription. The objections general aren't about
the availability (or lack thereof) of Windows build tools (especially
now the Python 2.7 compiler availability issue has been addressed via
http://www.microsoft.com/en-us/download/details.aspx?id=44266), it's
more about:

* wanting to build for Windows within the scope of an existing POSIX
based build automation system
* folks that prefer to use only open source software themselves
wanting to make their projects available to Windows users

"Python as first language" is still a relatively novel phenomenon, and
the existing distribution infrastructure certainly wasn't originally
designed for that use case.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 10/29/2014 10:31 AM, R. David Murray wrote:

> If you are writing code targeted for Windows, I think you are very 
> likely to have an MSDN subscription of some sort if your package
> includes C code.  I'm sure it's not 100%, though.

My experience with distributing distributions-with-extensions indicates
that the vast majority of Windows developers who are "downstream" users
for those distributions *cannot* build them from source:  if there is no
MSI / bdist_win (maybe now wheel), they won't use the project.

(Note that "having an MSDN subscription" is not the same as "knowing how
to configure which compiler such that it can bulid extensions against an
installed Python binary").


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlRRAskACgkQ+gerLs4ltQ7ILQCfbFCmgZqH+mZa28bQwjNuZruK
6BcAoLG/fxhi4LBkAgZoXNaxq6gi+Pbx
=8OvV
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Nick Coghlan]

(Paul Moore already covered most of this, but I'll go into a bit more
detail in a couple of areas)

On 29 October 2014 00:46, Tony Kelman  wrote:
> Stephen J. Turnbull:
>> It should be evident by now that our belief is that the large majority
>> of Windows users is well-served by the current model
>
>
> This is not the case at all in the scientific community. NumPy and SciPy
> put in a lot of extra work to come up with something that is compatible
> with the MSVC build of CPython because they have to, not because they're
> "happy to" jump through the necessary hoops.

Lots of folks are happy with POSIX emulation layers on Windows, as
they're OK with "basically works" rather than "works like any other
native application". "Basically works" isn't sufficient for many
Python-on-Windows use cases though, so the core ABI is a platform
native one, rather than a POSIX emulation.

This makes Python fit in more cleanly with other Windows applications,
but makes it harder to write Python applications that span both POSIX
and Windows. The degree to which a language runtime abstracts away the
details of the underlying operating system is a complex trade-off,
from C (where the C/C++ ABI is a core part of what *defines* an
operating system), through Python (which lets the developer largely
choose whether to use high level abstractions or lower level platform
specific APIs) to the JVM and CLR (which both largely abstract away
many of the details of the underlying operating system, aside from
carefully contained "native" code snippets).

This approach *does* fragment the community a bit, into at least
"Python-on-POSIX-only", "Python-on-Windows-only",
"platform-independent-Python" and "Python-on-POSIX-and-Windows" (where
the latter code has to deal directly with the differences between
Windows and POSIX because it needs the low level platform specific
APIs).

I personally consider that trade-off worth it, as it gives Python a
scope of applicability that more isolated runtimes lack, as well as
providing access to sophisticated platform level capabilities far more
cheaply than would be possible with a higher level of default
isolation.

> The situation today with
> NumPy AIUI is they already have to build with a custom hacked MinGW
> toolchain that only one person knows how to use. Evidently until very
> recently they were using a many-years-old 32-bit-only build of GCC 3.x.
> Do python-dev and numpy-discussion not talk to one another? I get that
> not everyone uses numpy, or Windows, but it sounds like there's very
> little understanding, or even awareness, of the issues they face.

There have been major ongoing communication problems stemming from the
wildly different assumptions that go into "programming as a tool for
building applications" (what python-dev and distutils-sig mostly do)
and "programming as a tool for thinking" (what scientists and data
analysts mostly do). Adding FORTRAN dependencies to the mix (which
*none* of the platform vendors really support properly) then makes the
whole problem of dealing with scientific tools even more alien to most
professional developers.

Those differing assumptions then meant that the two groups end up
frequently talk past each other because the worldviews are too
different. While that's starting to improve of late (as the above
distinction becomes better recognised), there's still a long way to
go.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Wed, 29 Oct 2014 11:07:53 -0400
Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 10/29/2014 10:31 AM, R. David Murray wrote:
> 
> > If you are writing code targeted for Windows, I think you are very 
> > likely to have an MSDN subscription of some sort if your package
> > includes C code.  I'm sure it's not 100%, though.
> 
> My experience with distributing distributions-with-extensions indicates
> that the vast majority of Windows developers who are "downstream" users
> for those distributions *cannot* build them from source:  if there is no
> MSI / bdist_win (maybe now wheel), they won't use the project.
> 
> (Note that "having an MSDN subscription" is not the same as "knowing how
> to configure which compiler such that it can bulid extensions against an
> installed Python binary").

I don't think you have to configure anything. Just install the right
Visual Studio version and it's done.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Thu, 30 Oct 2014 01:09:45 +1000
Nick Coghlan  wrote:
> 
> Lots of folks are happy with POSIX emulation layers on Windows, as
> they're OK with "basically works" rather than "works like any other
> native application". "Basically works" isn't sufficient for many
> Python-on-Windows use cases though, so the core ABI is a platform
> native one, rather than a POSIX emulation.
> 
> This makes Python fit in more cleanly with other Windows applications,
> but makes it harder to write Python applications that span both POSIX
> and Windows.

I don't really understanding why that's the case. Only the
building and packaging may be more difficult, and that assumes you're
familiar with mingw32. But mingw32, AFAIK, doesn't make the Windows
runtime magically POSIX-compatible (Cygwin does, to some extent).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Nathaniel Smith]

On 29 Oct 2014 14:47, "Antoine Pitrou"  wrote:
>
> On Wed, 29 Oct 2014 10:31:50 -0400
> "R. David Murray"  wrote:
>
> > On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver 
wrote:
> > > On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
> > >
> > > > most developers on Windows do have access to Microsoft tool
> > >
> > > I assume you mean python-dev folks who work on Windows:  it is
certainly
> > > not true for the vast majority of develoeprs who use Python on
Windows,
> > > who don't have the toolchain build their own C extensions.
> >
> > I'm pretty sure he meant "most people who develop software for Windows",
> > even though that's not how he phrased it.  But this does not include, as
> > you point out, people who develop Python software that *also* works on
> > Windows.
> >
> > If you are writing code targeted for Windows, I think you are very
> > likely to have an MSDN subscription of some sort if your package
includes C
> > code.  I'm sure it's not 100%, though.
>
> You can use Express editions of Visual Studio.

IIUC, the express edition compilers are 32-bit only, and what you actually
want are the "SDK compilers":
https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows

These are freely downloadable by anyone, no msdn subscription required, but
only if you know where to find them!

AFAICT the main obstacle to using MSVC to build python extensions (assuming
it can handle your code at all) is not anything technical, but rather that
there's no clear and correct tutorial on how to do it, and lots of
confusion and misinformation circulating.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Thu, 30 Oct 2014 01:09:45 +1000, Nick Coghlan  wrote:
> (Paul Moore already covered most of this, but I'll go into a bit more
> detail in a couple of areas)
> 
> On 29 October 2014 00:46, Tony Kelman  wrote:
> > Stephen J. Turnbull:
> >> It should be evident by now that our belief is that the large majority
> >> of Windows users is well-served by the current model
> >
> >
> > This is not the case at all in the scientific community. NumPy and SciPy
> > put in a lot of extra work to come up with something that is compatible
> > with the MSVC build of CPython because they have to, not because they're
> > "happy to" jump through the necessary hoops.
> 
> Lots of folks are happy with POSIX emulation layers on Windows, as
> they're OK with "basically works" rather than "works like any other
> native application". "Basically works" isn't sufficient for many
> Python-on-Windows use cases though, so the core ABI is a platform
> native one, rather than a POSIX emulation.

Since some of the context here is scientific use of Python, it might be
a useful bit of perspective to know that, while there are doubtless many
scientists using windows and using the windows native interfaces
happily, the Software Carpentry bootcamps that aim to give scientists
the basic framework for making better use of computers and software and
programming have as one foundation the bash shell, taught on Windows via
git-bash.  That is, the common toolset being taught to scientists (by
Software Carpentry) is the posix one, even on Windows.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Antoine Pitrou wrote:
> On Wed, 29 Oct 2014 11:07:53 -0400
> Tres Seaver  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>> 
>> On 10/29/2014 10:31 AM, R. David Murray wrote:
>> 
>> > If you are writing code targeted for Windows, I think you are very 
>> > likely to have an MSDN subscription of some sort if your package 
>> > includes C code.  I'm sure it's not 100%, though.
>> 
>> My experience with distributing distributions-with-extensions 
>> indicates that the vast majority of Windows developers who are 
>> "downstream" users for those distributions *cannot* build them from 
>> source:  if there is no MSI / bdist_win (maybe now wheel), they won't use 
>> the project.
>>
>> (Note that "having an MSDN subscription" is not the same as "knowing 
>> how to configure which compiler such that it can bulid extensions 
>> against an installed Python binary").
> 
> I don't think you have to configure anything. Just install the right Visual 
> Studio version and it's done.

The tricky part here is the *right* Visual Studio version. As many have noted, 
there were bugs/missing components in some of the older Express editions (like 
the 64-bit compiler integration), and even the compiler pack we released 
recently doesn't actually help building Python itself, though it's good for 
extensions. In general, VS 2008 Professional and VS 2010 Professional are 
easiest to set up if you can get them, the C++ Express editions are fairly easy 
to set up, and the Windows SDK is difficult but possible (and won't currently 
build CPython either, as the build depends on VS - I'm also fixing this for 
3.5).

My ideal target (Utopia refined to be achievable) is for python-dev to handle 
the Python binaries themselves (already done) and to make them easy to bundle 
with applications/etc (I'm working on this for 3.5), and for PyPI to include 
pre-built wheels for Windows where necessary.

Note that I don't require package developers to build their own wheels, though 
I think that they are generally the right people to be doing it. It would be 
nice to create a culture of delegation, so that "someone" could be a trusted 
builder for a range of packages (for example, I'd love it if 
Continuum/Enthought/similar could provide their builds of numpy/scipy as wheels 
and those projects would be willing to have them be the official PyPI 
downloads). This is roughly how the python.org installers are handled, and 
while it may cause some lag between source and binary releases, I think the 
release cycles are slow enough that most users would only notice that "pip 
install numpy" now works.

Cheers,
Steve

> Regards
> 
> Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[David Cournapeau]

On Wed, Oct 29, 2014 at 3:25 PM, Antoine Pitrou  wrote:

> On Thu, 30 Oct 2014 01:09:45 +1000
> Nick Coghlan  wrote:
> >
> > Lots of folks are happy with POSIX emulation layers on Windows, as
> > they're OK with "basically works" rather than "works like any other
> > native application". "Basically works" isn't sufficient for many
> > Python-on-Windows use cases though, so the core ABI is a platform
> > native one, rather than a POSIX emulation.
> >
> > This makes Python fit in more cleanly with other Windows applications,
> > but makes it harder to write Python applications that span both POSIX
> > and Windows.
>
> I don't really understanding why that's the case. Only the
> building and packaging may be more difficult, and that assumes you're
> familiar with mingw32. But mingw32, AFAIK, doesn't make the Windows
> runtime magically POSIX-compatible (Cygwin does, to some extent).
>

mingw32 is a more compliant C compiler (VS2008 does not implement much from
C89), and it does implement quite a few things not implemented in the C
runtime, especially for math.

But TBH, those are not compelling cases to build python itself on mingw,
only to better support C extensions with mingw.

David


> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cournape%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[David Cournapeau]

On Wed, Oct 29, 2014 at 5:17 PM, David Cournapeau 
wrote:

>
>
> On Wed, Oct 29, 2014 at 3:25 PM, Antoine Pitrou 
> wrote:
>
>> On Thu, 30 Oct 2014 01:09:45 +1000
>> Nick Coghlan  wrote:
>> >
>> > Lots of folks are happy with POSIX emulation layers on Windows, as
>> > they're OK with "basically works" rather than "works like any other
>> > native application". "Basically works" isn't sufficient for many
>> > Python-on-Windows use cases though, so the core ABI is a platform
>> > native one, rather than a POSIX emulation.
>> >
>> > This makes Python fit in more cleanly with other Windows applications,
>> > but makes it harder to write Python applications that span both POSIX
>> > and Windows.
>>
>> I don't really understanding why that's the case. Only the
>> building and packaging may be more difficult, and that assumes you're
>> familiar with mingw32. But mingw32, AFAIK, doesn't make the Windows
>> runtime magically POSIX-compatible (Cygwin does, to some extent).
>>
>
> mingw32 is a more compliant C compiler (VS2008 does not implement much
> from C89)
>

That should read much C99, of course, otherwise VS 2008 would have been a
completely useless C compiler !

David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

> On Oct 29, 2014, at 11:37 AM, Steve Dower  wrote:
> 
> Antoine Pitrou wrote:
>> On Wed, 29 Oct 2014 11:07:53 -0400
>> Tres Seaver  wrote:
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA1
>>> 
>>> On 10/29/2014 10:31 AM, R. David Murray wrote:
>>> 
 If you are writing code targeted for Windows, I think you are very 
 likely to have an MSDN subscription of some sort if your package 
 includes C code.  I'm sure it's not 100%, though.
>>> 
>>> My experience with distributing distributions-with-extensions 
>>> indicates that the vast majority of Windows developers who are 
>>> "downstream" users for those distributions *cannot* build them from 
>>> source:  if there is no MSI / bdist_win (maybe now wheel), they won't use 
>>> the project.
>>> 
>>> (Note that "having an MSDN subscription" is not the same as "knowing 
>>> how to configure which compiler such that it can bulid extensions 
>>> against an installed Python binary").
>> 
>> I don't think you have to configure anything. Just install the right Visual 
>> Studio version and it's done.
> 
> The tricky part here is the *right* Visual Studio version. As many have 
> noted, there were bugs/missing components in some of the older Express 
> editions (like the 64-bit compiler integration), and even the compiler pack 
> we released recently doesn't actually help building Python itself, though 
> it's good for extensions. In general, VS 2008 Professional and VS 2010 
> Professional are easiest to set up if you can get them, the C++ Express 
> editions are fairly easy to set up, and the Windows SDK is difficult but 
> possible (and won't currently build CPython either, as the build depends on 
> VS - I'm also fixing this for 3.5).
> 
> My ideal target (Utopia refined to be achievable) is for python-dev to handle 
> the Python binaries themselves (already done) and to make them easy to bundle 
> with applications/etc (I'm working on this for 3.5), and for PyPI to include 
> pre-built wheels for Windows where necessary.
> 
> Note that I don't require package developers to build their own wheels, 
> though I think that they are generally the right people to be doing it. It 
> would be nice to create a culture of delegation, so that "someone" could be a 
> trusted builder for a range of packages (for example, I'd love it if 
> Continuum/Enthought/similar could provide their builds of numpy/scipy as 
> wheels and those projects would be willing to have them be the official PyPI 
> downloads). This is roughly how the python.org installers are handled, and 
> while it may cause some lag between source and binary releases, I think the 
> release cycles are slow enough that most users would only notice that "pip 
> install numpy" now works.

For the record, a long term “down the road” kind of thing I want to do is have 
PyPI have a build farm which can build Wheels. So that people upload a source 
distribution to PyPI and it kicks off Wheel builds on the various platforms 
automatically.

What this looks like is a complete unknown, all I have is the general idea.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Impact of Windows PowerShell OneGet ?

[Glenn Linderman]

New package manager from M$... article here 
.


It seems doubtful that M$ will eliminate .msi (their obscure, hard to 
configure and use, installation format), so it seems doubtful that the 
addition of OneGet will _force_ any changes to Python packaging.


However, it does open the question in my mind about whether there will 
be any _benefits_ of OneGet that would inspire helpful, useful changes 
to Python packaging. They speak of "trusted repositories", and the like, 
and it sounds like a the various *nix package managers (apt-get, et 
alia), but perhaps allowing multiple repositories rather than just a 
single source vendor repository (I'm actually not sure if *nix package 
managers allow multiple repositories or not, but from the way people 
talk about them, it always sounds like a "distribution" also provides "a 
repository" of additional packages).


"trusted repositories" sounds more like Perl's CPAN.

One of the links 
 
contains this quote: "This first version of OneGet installs and searches 
software from Chocolatey repositories.  Support of additional 
repositories will come in subsequent versions."


I have no clue what a Chocolatey repository is (yet, will Google), but 
unknown others will come, it says... whether it is possible to write a 
"repository plugin" such that Perl's CPAN or Python's PyPI or other 
preexisting repositories can be accessed is not clear.


The relationship between PowerShell and OneGet is not clear either... is 
OneGet written in PowerShell, or is PowerShell just one way to invoke 
OneGet, or???


Just a heads up.






___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Impact of Windows PowerShell OneGet ?

[Donald Stufft]

> On Oct 29, 2014, at 3:34 PM, Glenn Linderman  wrote:
> 
> New package manager from M$... article here 
> .
> 
> It seems doubtful that M$ will eliminate .msi (their obscure, hard to 
> configure and use, installation format), so it seems doubtful that the 
> addition of OneGet will _force_ any changes to Python packaging.
> 
> However, it does open the question in my mind about whether there will be any 
> _benefits_ of OneGet that would inspire helpful, useful changes to Python 
> packaging. They speak of "trusted repositories", and the like, and it sounds 
> like a the various *nix package managers (apt-get, et alia), but perhaps 
> allowing multiple repositories rather than just a single source vendor 
> repository (I'm actually not sure if *nix package managers allow multiple 
> repositories or not, but from the way people talk about them, it always 
> sounds like a "distribution" also provides "a repository" of additional 
> packages).
> 
> "trusted repositories" sounds more like Perl's CPAN.
> 
> One of the links 
> 
>  contains this quote: "This first version of OneGet installs and searches 
> software from Chocolatey repositories.  Support of additional repositories 
> will come in subsequent versions."
> 
> I have no clue what a Chocolatey repository is (yet, will Google), but 
> unknown others will come, it says... whether it is possible to write a 
> "repository plugin" such that Perl's CPAN or Python's PyPI or other 
> preexisting repositories can be accessed is not clear.
> 
> The relationship between PowerShell and OneGet is not clear either... is 
> OneGet written in PowerShell, or is PowerShell just one way to invoke OneGet, 
> or???
> 
> Just a heads up.
> 

It appears to be a package manager manager. Chocolatey is one of the third 
party package managers available on Windows.

I also just learned that OneGet is apparently OSS and developed on github 
(https://github.com/OneGet/oneget ).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 29 October 2014 15:31, Nathaniel Smith  wrote:
>> You can use Express editions of Visual Studio.
>
> IIUC, the express edition compilers are 32-bit only, and what you actually
> want are the "SDK compilers":
> https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
>
> These are freely downloadable by anyone, no msdn subscription required, but
> only if you know where to find them!
>
> AFAICT the main obstacle to using MSVC to build python extensions (assuming
> it can handle your code at all) is not anything technical, but rather that
> there's no clear and correct tutorial on how to do it, and lots of confusion
> and misinformation circulating.

Would it help if I wrote a document explaining how to set up the MS
compilers (free and paid for) to allow building of Python extensions?

There are a few provisos:

1. A lot of it will be pretty trivial: "If you have Vistal Studio
(full edition), install it. Done."
2. It will be out of date very fast as the situation for Python 3.5+
will be trivial across the board.
3. I don't have anywhere particularly authoritative to host it (maybe
the Python Wiki?) and it could easily get lost in the huge swamp of
variously outdated, over-complicated, or otherwise alternative
documents available. Ideally I'd like someone to suggest an "official"
location I could use.

I don't want to do this if it won't be useful, as it'll take me a bit
of effort to confirm the process for the only non-trivial scenario
(64-bit Python 3.3/3.4 with free tools). But if people think it would
help, that's fine, I volunteer.

Paul

PS Even if I don't get positive feedback, I may just say "to heck with
it" and do it anyway, because it *is* so trivial :-) I just won't
promise.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Tony Kelman]

Stephen J. Turnbull:

the pain of using Windows is what drives me away from all of them.


Enough that you are not able to make the software you write usable on
Windows? I see your point and agree with it - I don't even like Windows
much at all, but supporting it is important for plenty of reasons.

Steve Dower:

FWIW, I'm happy to talk specifics off list


Off-list, on-list, either way. I've read the blog post about the CRT
refactoring 
http://blogs.msdn.com/b/vcblog/archive/2014/06/10/the-great-crt-refactoring.aspx

but have not yet experimented with the early CTP versions of Visual Studio
"14." Is there any plan to promote these runtimes to installed-by-default
components of the operating system in future service packs or new releases
of Windows? The fact that the redistributables are a separate download and
not always there by default means MinGW developers are not quite so
optimistic about them suddently solving all problems. In the embedded-
Python-for-IJulia pull request on github for example, it turns out the
Python msi installer does not include the necessary runtimes and wouldn't
work by itself on a brand new computer.

It would also help if there were a preview version of just the runtime
libraries available. More far-fetched and not likely to happen would be a
smaller "command line tools for Visual Studio" type package, like Apple
provides for Xcode, with only the compiler, tools, and libraries needed
to build C/C++ libraries or Python extensions but not the whole IDE.

Nick Coghlan:

* wanting to build for Windows within the scope of an existing POSIX
based build automation system
* folks that prefer to use only open source software themselves
wanting to make their projects available to Windows users


Don't forget: * MSVC is not capable of compiling the code I need to use.
That's the crux of the scientific-software problem. If it were possible to
use MSVC for everything, I'd probably suck it up and spend my time writing
CMake build systems for every project I wanted to get to work on Windows.
Most software I work with is a library first, Python extension second, if
at all.

Nick Coghlan:

Lots of folks are happy with POSIX emulation layers on Windows, as
they're OK with "basically works" rather than "works like any other
native application". "Basically works" isn't sufficient for many
Python-on-Windows use cases though, so the core ABI is a platform
native one, rather than a POSIX emulation.


To clarify here: MinGW does not use a POSIX layer at all. Cygwin does,
MSYS does (it's based on Cygwin with relatively minor changes), but when
we're discussing the MinGW.org or MinGW-w64 compilers, there is no POSIX.
There is a GCC runtime library, which can be linked statically if you
choose, though it's not always a good idea to do so. Cygwin or MSYS provide
a POSIX build environment where you can run bash, make, etc, but with MinGW
you are not targeting or depending on that build environment with the
compiled binaries.

The distinction in runtime libraries is that MinGW links to msvcrt.dll,
which is from an old version of Visual Studio but is installed by default
on every version of Windows since XP. The python.org installers are
compiled with a particular more recent version of Visual Studio, so
they (and hence all extensions used with them) depend on the specific
corresponding msvcr##.dll.

Most MinGW developers and users would rather avoid the dll hell and issues
stemming from choosing a particular MSVC runtime and having to stick with
it for everything. NumPy was able to coax a custom MinGW-w64 build into
linking to CPython's designated runtime and statically link all the GCC
libraries (these are 2 separate issues however). I do think some sensitivity
to the requirements of the existing Windows CPython/PyPI environment from
the MinGW-w64 side would be valuable. If NumPy's needs can be met with
fewer modifications and without having to rebuild a custom GCC from source,
I think that would be beneficial for everyone. We'll see how many of the
MinGW-w64 developers I can get to agree with me on that. Any changes that
may need to be upstreamed to GCC will have their own set of challenges.


Adding FORTRAN dependencies to the mix (which
*none* of the platform vendors really support properly)


I'm not sure what you mean by platform vendors? Are you talking Conda,
Enthought, etc? To the best of my knowledge, they're using the Intel
toolchain for Fortran, at least on Windows and probably elsewhere. So
I suspect you'll find a libifcoremd.dll somewhere in those distributions.
That imposes some cost requirements on reproducing their processes that
are a bit high for average users to meet.

Donald Stufft:

So that people upload a source distribution to PyPI and it kicks off
Wheel builds on the various platforms automatically.

What this looks like is a complete unknown, all I have is the general
idea.


I do something similar today for C, C++, and Fortran libraries using
the MinGW-w64 cross-compiler. It loo

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

This sounds like something good for packaging.python.org


> On Oct 29, 2014, at 4:05 PM, Paul Moore  wrote:
> 
> On 29 October 2014 15:31, Nathaniel Smith  wrote:
>>> You can use Express editions of Visual Studio.
>> 
>> IIUC, the express edition compilers are 32-bit only, and what you actually
>> want are the "SDK compilers":
>> https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
>> 
>> These are freely downloadable by anyone, no msdn subscription required, but
>> only if you know where to find them!
>> 
>> AFAICT the main obstacle to using MSVC to build python extensions (assuming
>> it can handle your code at all) is not anything technical, but rather that
>> there's no clear and correct tutorial on how to do it, and lots of confusion
>> and misinformation circulating.
> 
> Would it help if I wrote a document explaining how to set up the MS
> compilers (free and paid for) to allow building of Python extensions?
> 
> There are a few provisos:
> 
> 1. A lot of it will be pretty trivial: "If you have Vistal Studio
> (full edition), install it. Done."
> 2. It will be out of date very fast as the situation for Python 3.5+
> will be trivial across the board.
> 3. I don't have anywhere particularly authoritative to host it (maybe
> the Python Wiki?) and it could easily get lost in the huge swamp of
> variously outdated, over-complicated, or otherwise alternative
> documents available. Ideally I'd like someone to suggest an "official"
> location I could use.
> 
> I don't want to do this if it won't be useful, as it'll take me a bit
> of effort to confirm the process for the only non-trivial scenario
> (64-bit Python 3.3/3.4 with free tools). But if people think it would
> help, that's fine, I volunteer.
> 
> Paul
> 
> PS Even if I don't get positive feedback, I may just say "to heck with
> it" and do it anyway, because it *is* so trivial :-) I just won't
> promise.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 29 October 2014 20:26, Donald Stufft  wrote:
> This sounds like something good for packaging.python.org

Yeah, I wondered about that. I'll work up a patch for that. But the
more I think about it, it really is trivial:

- For non-free MSVC, install the appropriate version, and everything just works.
- For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
package and everything just works as long as you're using setuptools.
- For 32 bit Python 3.2-3.4, install Visual Studio Express and
everything just works.
- For 64 bit Python 3.2-3.4, install the SDK, set some environment
variables, and everything just works.
- For Python 3.5+, install the current Visual Studion Express and
everything just works.

(I propose to deem Python 2.7 without setuptools as "unsupported")

The only things I can see that need expansion are:

1. The precise versions to use (trivial to add, I'm just to lazy to
check right now).
2. Where to get VS 2010 Express (as it's no longer supported or
available from MS).
3. How to set up the SDK environment for 64-bit Python 3.2-3.4.

Before I do a write-up, I want to set up clean VMs with these
configurations, so that I can confirm the details. But I don't expect
any problems, as I've done them all before.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

> On Oct 29, 2014, at 6:09 PM, Paul Moore  wrote:
> 
> On 29 October 2014 20:26, Donald Stufft  wrote:
>> This sounds like something good for packaging.python.org
> 
> Yeah, I wondered about that. I'll work up a patch for that. But the
> more I think about it, it really is trivial:
> 
> - For non-free MSVC, install the appropriate version, and everything just 
> works.
> - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
> package and everything just works as long as you're using setuptools.
> - For 32 bit Python 3.2-3.4, install Visual Studio Express and
> everything just works.
> - For 64 bit Python 3.2-3.4, install the SDK, set some environment
> variables, and everything just works.
> - For Python 3.5+, install the current Visual Studion Express and
> everything just works.
> 
> (I propose to deem Python 2.7 without setuptools as "unsupported")
> 
> The only things I can see that need expansion are:
> 
> 1. The precise versions to use (trivial to add, I'm just to lazy to
> check right now).
> 2. Where to get VS 2010 Express (as it's no longer supported or
> available from MS).
> 3. How to set up the SDK environment for 64-bit Python 3.2-3.4.
> 
> Before I do a write-up, I want to set up clean VMs with these
> configurations, so that I can confirm the details. But I don't expect
> any problems, as I've done them all before.
> 
> Paul.

I think it’d be good even if considered trivial. I know I certainly have no
idea which pieces I needed to do that.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Ethan Furman]

On 10/29/2014 03:09 PM, Paul Moore wrote:

On 29 October 2014 20:26, Donald Stufft  wrote:

This sounds like something good for packaging.python.org


Yeah, I wondered about that. I'll work up a patch for that. But the
more I think about it, it really is trivial:


I am reminded of an interview question I was once asked which was prefaced with: 
"Here's an easy one..."

My reply was, if you know the answer, it's easy!



- For non-free MSVC, install the appropriate version, and everything just works.
- For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
package and everything just works as long as you're using setuptools.
- For 32 bit Python 3.2-3.4, install Visual Studio Express and
everything just works.
- For 64 bit Python 3.2-3.4, install the SDK, set some environment
variables, and everything just works.
- For Python 3.5+, install the current Visual Studion Express and
everything just works.


I would suggest
  - where to get these packages
  - where to get any dependencies
  - any options to [not] specify during install
  - what environment variables to what value
  - where one should be at when one starts the compile process

and, of course, a gotchas section for uncommon but frustrating things that 
might go wrong.

And thanks for doing this!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 29 October 2014 22:19, Ethan Furman  wrote:
>> Yeah, I wondered about that. I'll work up a patch for that. But the
>> more I think about it, it really is trivial:
>
> I am reminded of an interview question I was once asked which was prefaced
> with: "Here's an easy one..."
>
> My reply was, if you know the answer, it's easy!

Yeah, I know what you mean. My take on this is that I agree it's not
easy if you don't know and can't get access to the information, but if
you can, there's very little to it.

>> - For non-free MSVC, install the appropriate version, and everything just
>> works.
>> - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
>> package and everything just works as long as you're using setuptools.
>> - For 32 bit Python 3.2-3.4, install Visual Studio Express and
>> everything just works.
>> - For 64 bit Python 3.2-3.4, install the SDK, set some environment
>> variables, and everything just works.
>> - For Python 3.5+, install the current Visual Studion Express and
>> everything just works.
>
>
> I would suggest
>   - where to get these packages

Conceded. Working out how to point people at stuff on MSDN is a
challenge, things seem to move around. Maybe Steve Dower could help
here with canonical URLs for some of them (IIRC, he provided one for
the VS compilers for Python 2.7 package when it was announced).

For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)

And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly.

>   - where to get any dependencies

There are none. I could state that explicitly, I guess.

>   - any options to [not] specify during install

I'll have to go through the installs again just to be sure, but I'm
pretty certain "Select the default for everything" is correct.

>   - what environment variables to what value

None, except for the SDK which I did say I needed to test out and
cover in more detail.

>   - where one should be at when one starts the compile process

I don't understand this. It's just "pip wheel foo" to build a wheel
for foo (which will be downloaded), or "pip wheel ." or  "python
setup.py bdist_wheel" as you prefer for a local package. That's
standard distutils/setuptools/pip extension building. I don't propose
to cover that, just how to *set up* the environment.

With the sole exception of the SDK case, once installed, everything
just works everywhere, nothing to set up, no special environment to
configure. Start up a new cmd or powershell console (or use the one
you're already in) and go. Maybe Unix users expect more complexity
because it's not that simple on Unix? But I thought it was - install
the appropriate OS packages and that's it?

> and, of course, a gotchas section for uncommon but frustrating things that
> might go wrong.

Hmm, I see your point here, but I'm not sure what I might include. You
*can* get in a mess [1] but generally I don't as I'm an experienced
Windows user. I also don't want to offer to debug and fix everyone's
problems in getting things set up, so offering to collect and document
"common issues" that I've seen is impractical. Maybe a section
entitled "Common Issues and Their Solutions", with some placeholder
text saying "If you have any issues installing any of the compiler
packages mentioned, please document what went wrong and how to fix it,
and submit a PR!" would do?

> And thanks for doing this!

No problem!

Paul

[1] I once spent a *long* time fighting failed installs of the Windows
SDK. Turns out it was because I was trying to install a 32-bit SDK on
a 64-bit machine (doh!), and the installer really doesn't like that
:-( About all I could say to document that is "Read the instructions
properly" and "I'm sorry that the MS installers fail really badly when
faced with relatively obvious idiot-user errors, but I can't do
anything about it :-("
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Ethan Furman]

On 10/29/2014 03:46 PM, Paul Moore wrote:

On 29 October 2014 22:19, Ethan Furman  wrote:


   - where one should be at when one starts the compile process


I don't understand this. It's just "pip wheel foo" to build a wheel
for foo (which will be downloaded), or "pip wheel ." or  "python
setup.py bdist_wheel" as you prefer for a local package.


Hmmm...  That looks like it's for installing/compiling somebody else's package.  Is that last command sufficient to 
prepare one's own wheel for uploading to PyPI, or there something else to do?


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Donald Stufft]

> On Oct 29, 2014, at 7:02 PM, Ethan Furman  wrote:
> 
> On 10/29/2014 03:46 PM, Paul Moore wrote:
>> On 29 October 2014 22:19, Ethan Furman  wrote:
>>> 
>>>   - where one should be at when one starts the compile process
>> 
>> I don't understand this. It's just "pip wheel foo" to build a wheel
>> for foo (which will be downloaded), or "pip wheel ." or  "python
>> setup.py bdist_wheel" as you prefer for a local package.
> 
> Hmmm...  That looks like it's for installing/compiling somebody else's 
> package.  Is that last command sufficient to prepare one's own wheel for 
> uploading to PyPI, or there something else to do?
> 

Generally for uploading to PyPI you do ``python setup.py bdist_wheel``, though 
I don’t think there’d be any bad thing if you used pip wheel.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 29 October 2014 23:02, Ethan Furman  wrote:
> On 10/29/2014 03:46 PM, Paul Moore wrote:
>>
>> On 29 October 2014 22:19, Ethan Furman  wrote:
>>>
>>>
>>>- where one should be at when one starts the compile process
>>
>>
>> I don't understand this. It's just "pip wheel foo" to build a wheel
>> for foo (which will be downloaded), or "pip wheel ." or  "python
>> setup.py bdist_wheel" as you prefer for a local package.
>
>
> Hmmm...  That looks like it's for installing/compiling somebody else's
> package.  Is that last command sufficient to prepare one's own wheel for
> uploading to PyPI, or there something else to do?

Oh, I see what you're thinking.

I explicitly *don't* want to get into general packaging issues - they
are covered elsewhere. The point here is that if you have the right
compiler set up, you can read any generic packaging instructions and
it doesn't matter whether there's C code involved. That's it.

But yes, "python setup.py bdist_wheel" is how you build a wheel for
upload. That's true whether your project includes C extensions or not.

I'm also not expecting to explain to people how to build any
dependencies (for example, PyYAML depends on the C compiler being able
to find libyaml). That *is* harder on Windows than on Linux, because
there's no "system location" equivalent to /usr/local/include and
/usr/local/lib. But again, I consider this to be outside the scope of
the document, because it's specific to the particular project you're
building.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Nathaniel Smith]

On Wed, Oct 29, 2014 at 10:46 PM, Paul Moore  wrote:
> On 29 October 2014 22:19, Ethan Furman  wrote:
>>> Yeah, I wondered about that. I'll work up a patch for that. But the
>>> more I think about it, it really is trivial:
>>
>> I am reminded of an interview question I was once asked which was prefaced
>> with: "Here's an easy one..."
>>
>> My reply was, if you know the answer, it's easy!
>
> Yeah, I know what you mean. My take on this is that I agree it's not
> easy if you don't know and can't get access to the information, but if
> you can, there's very little to it.

That's great, but yeah. In case it helps as a data point, I consider
myself a reasonably technical guy, probably more than your average
python package author -- 15 years of free software hacking, designed a
pretty popular X remote display protocol, helped invent DVCS, have
written patches to gcc and the kernel, numpy co-maintainer, etc. etc.

For the last ~12 months I've had an underlined and circled todo item
saying "make wheels for https://pypi.python.org/pypi/zs/";, and every
time I've sat down and spent a few hours trying I've ended up utterly
defeated with a headache. Distutils is spaghetti, and Windows is
spaghetti (esp. if you've never used it for more than checking email
on a friend's laptop), and the toolchain setup is spaghetti, and then
suddenly people are talking about vcvarsall.bats and I don't know what
all. I honestly would totally benefit from one of those
talk-your-grandparents-through-it tutorials ("here's a screenshot of
the dialogue box, and I've drawn an arrow pointing at the 'ok' button.
You should press the 'ok' button.")

>>> - For non-free MSVC, install the appropriate version, and everything just
>>> works.
>>> - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
>>> package and everything just works as long as you're using setuptools.
>>> - For 32 bit Python 3.2-3.4, install Visual Studio Express and
>>> everything just works.
>>> - For 64 bit Python 3.2-3.4, install the SDK, set some environment
>>> variables, and everything just works.

I think the SDK covers both 32 and 64 bit? If so then leaving visual
studio out of things entirely is probably simpler.

I'm also unsure how you even get both 32- and 64-bit versions of
python to coexist on the same system.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 29 October 2014 23:22, Nathaniel Smith  wrote:
>> Yeah, I know what you mean. My take on this is that I agree it's not
>> easy if you don't know and can't get access to the information, but if
>> you can, there's very little to it.
>
> That's great, but yeah. In case it helps as a data point, I consider
> myself a reasonably technical guy, probably more than your average
> python package author -- 15 years of free software hacking, designed a
> pretty popular X remote display protocol, helped invent DVCS, have
> written patches to gcc and the kernel, numpy co-maintainer, etc. etc.

Yeah, that counts :-)

> For the last ~12 months I've had an underlined and circled todo item
> saying "make wheels for https://pypi.python.org/pypi/zs/";, and every
> time I've sat down and spent a few hours trying I've ended up utterly
> defeated with a headache. Distutils is spaghetti, and Windows is
> spaghetti (esp. if you've never used it for more than checking email
> on a friend's laptop), and the toolchain setup is spaghetti, and then
> suddenly people are talking about vcvarsall.bats and I don't know what
> all. I honestly would totally benefit from one of those
> talk-your-grandparents-through-it tutorials ("here's a screenshot of
> the dialogue box, and I've drawn an arrow pointing at the 'ok' button.
> You should press the 'ok' button.")

OK. That needs to be fixed. How about this. I'll work with you to get
the wheel build process set up in such a way that you can maintain it,
and we'll document what you needed to know in order to do it. Then I
can put that documentation into the packaging user guide for other
people to work with. It can wait till you have the time, but when you
do, shout and I'll help.

I know nothing about zs, but I just tried building it. MSVC 2010
doesn't support "static inline" (used twice in zs\pycrc-crc64xz.h) but
when I removed the "inline" from that it built fine for me. So it
should be a good example, as there are no particular complexities to
deal with.

 - For non-free MSVC, install the appropriate version, and everything just
 works.
 - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
 package and everything just works as long as you're using setuptools.
 - For 32 bit Python 3.2-3.4, install Visual Studio Express and
 everything just works.
 - For 64 bit Python 3.2-3.4, install the SDK, set some environment
 variables, and everything just works.
>
> I think the SDK covers both 32 and 64 bit? If so then leaving visual
> studio out of things entirely is probably simpler.

The SDK is the most complex environment to set up, so avoiding it
unless you need it is a good thing. But maybe it's worth explaining
that if you need it anyway, this is how you set it up to cover some of
the other cases as well.

> I'm also unsure how you even get both 32- and 64-bit versions of
> python to coexist on the same system.

Just install them into different directories. The default is the same
for 32 and 64 bit, so you need to override the default for (at least)
one of them, but that's all.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 29 October 2014 23:49, Steve Dower  wrote:
> "For the paid versions, I'm going to assume that anyone who paid for a
> compiler and doesn't know where their copy is, probably can't be
> helped ;-)"
>
> You could link to visualstudio.com for the trial versions, and maybe to a
> page/post about the PSF's grants process if such a page exists.

That doesn't help for VS 2010 and VS 2008, which are the relevant ones
right now, unfortunately. In thew brave new world of Python 3.5+ that
might well do, though. (Although the feedback from Unix users is that
they really want a direct link to the exact edition they need - i.e.,
a permalink to VC++ Express Edition  for Python 3.5+. I'm not
sure how well the MS website structure does that, though.

> "And of course there's the awkward problem that VS 2010 Express is
> unsupported and therefore no longer available from *any* official
> location. I can't solve that, sadly"
>
> These are still at
> http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#DownloadFamilies_4,
> which is the main download page. Hopefully they don't go away before 3.5,
> but I have no control over that unfortunately.

Wow! I didn't know that. That's great, solves the biggest issue I had.

> The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I
> own and can update if necessary.

Cool. As long as it's stable and reliable, it's exactly what I want.

Many thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

"For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)"

You could link to visualstudio.com for the trial versions, and maybe to a 
page/post about the PSF's grants process if such a page exists.

"And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly"

These are still at 
http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#DownloadFamilies_4,
 which is the main download page. Hopefully they don't go away before 3.5, but 
I have no control over that unfortunately.

The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I own 
and can update if necessary.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore
Sent: ‎10/‎29/‎2014 15:48
To: Ethan Furman
Cc: Python Dev
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 22:19, Ethan Furman  wrote:
>> Yeah, I wondered about that. I'll work up a patch for that. But the
>> more I think about it, it really is trivial:
>
> I am reminded of an interview question I was once asked which was prefaced
> with: "Here's an easy one..."
>
> My reply was, if you know the answer, it's easy!

Yeah, I know what you mean. My take on this is that I agree it's not
easy if you don't know and can't get access to the information, but if
you can, there's very little to it.

>> - For non-free MSVC, install the appropriate version, and everything just
>> works.
>> - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
>> package and everything just works as long as you're using setuptools.
>> - For 32 bit Python 3.2-3.4, install Visual Studio Express and
>> everything just works.
>> - For 64 bit Python 3.2-3.4, install the SDK, set some environment
>> variables, and everything just works.
>> - For Python 3.5+, install the current Visual Studion Express and
>> everything just works.
>
>
> I would suggest
>   - where to get these packages

Conceded. Working out how to point people at stuff on MSDN is a
challenge, things seem to move around. Maybe Steve Dower could help
here with canonical URLs for some of them (IIRC, he provided one for
the VS compilers for Python 2.7 package when it was announced).

For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)

And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly.

>   - where to get any dependencies

There are none. I could state that explicitly, I guess.

>   - any options to [not] specify during install

I'll have to go through the installs again just to be sure, but I'm
pretty certain "Select the default for everything" is correct.

>   - what environment variables to what value

None, except for the SDK which I did say I needed to test out and
cover in more detail.

>   - where one should be at when one starts the compile process

I don't understand this. It's just "pip wheel foo" to build a wheel
for foo (which will be downloaded), or "pip wheel ." or  "python
setup.py bdist_wheel" as you prefer for a local package. That's
standard distutils/setuptools/pip extension building. I don't propose
to cover that, just how to *set up* the environment.

With the sole exception of the SDK case, once installed, everything
just works everywhere, nothing to set up, no special environment to
configure. Start up a new cmd or powershell console (or use the one
you're already in) and go. Maybe Unix users expect more complexity
because it's not that simple on Unix? But I thought it was - install
the appropriate OS packages and that's it?

> and, of course, a gotchas section for uncommon but frustrating things that
> might go wrong.

Hmm, I see your point here, but I'm not sure what I might include. You
*can* get in a mess [1] but generally I don't as I'm an experienced
Windows user. I also don't want to offer to debug and fix everyone's
problems in getting things set up, so offering to collect and document
"common issues" that I've seen is impractical. Maybe a section
entitled "Common Issues and Their Solutions", with some placeholder
text saying "If you have any issues installing any of the compiler
packages mentioned, please document what went wrong and how to fix it,
and submit a PR!" would do?

> And thanks for doing this!

No problem!

Paul

[1] I once spent a *long* time fighting failed installs of the Windows
SDK. Turns out it was because I was trying to install a 32-bit SDK on
a 64-bit machine (doh!), and the installer really doesn't like that
:-( About all I could say to document that is "Read the instructions
properly" and "I'm 

Re: [Python-Dev] Status of C compilers for Python on Windows

[Terry Reedy]

On 10/29/2014 4:05 PM, Paul Moore wrote:

On 29 October 2014 15:31, Nathaniel Smith  wrote:

You can use Express editions of Visual Studio.


IIUC, the express edition compilers are 32-bit only, and what you actually
want are the "SDK compilers":
https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows

These are freely downloadable by anyone, no msdn subscription required, but
only if you know where to find them!

AFAICT the main obstacle to using MSVC to build python extensions (assuming
it can handle your code at all) is not anything technical, but rather that
there's no clear and correct tutorial on how to do it, and lots of confusion
and misinformation circulating.


Would it help if I wrote a document explaining how to set up the MS
compilers (free and paid for) to allow building of Python extensions?



There are a few provisos:

1. A lot of it will be pretty trivial: "If you have Vistal Studio
(full edition), install it. Done."
2. It will be out of date very fast as the situation for Python 3.5+
will be trivial across the board.
3. I don't have anywhere particularly authoritative to host it (maybe
the Python Wiki?) and it could easily get lost in the huge swamp of
variously outdated, over-complicated, or otherwise alternative
documents available. Ideally I'd like someone to suggest an "official"
location I could use.


There is already a section in the devguide for building Python itself, 
with mostly the same info, except it may not be up to date.



I don't want to do this if it won't be useful, as it'll take me a bit
of effort to confirm the process for the only non-trivial scenario
(64-bit Python 3.3/3.4 with free tools). But if people think it would
help, that's fine, I volunteer.


The devguide needs to be kept up to date.  If you open a tracker issue, 
put me as nosy to review and commit.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Terry Reedy]

On 10/29/2014 11:37 AM, Steve Dower wrote:


My ideal target (Utopia refined to be achievable) is for python-dev
to handle the Python binaries themselves (already done) and to make
them easy to bundle with applications/etc (I'm working on this for
3.5), and for PyPI to include pre-built wheels for Windows where
necessary.

Note that I don't require package developers to build their own
wheels, though I think that they are generally the right people to be
doing it. It would be nice to create a culture of delegation, so that
"someone" could be a trusted builder for a range of packages


Christoph Gohlke, Laboratory for Fluorescence Dynamics, University of 
California, Irvine now provides 32 and 64 bit builds for 2.6/7, 3.2/3/4 
(10 total) for about 300 'scientific open-source extension packages' an 
his site.

http://www.lfd.uci.edu/~gohlke/pythonlibs/

Perhaps the PyPI folks could discuss incorporating wheel versions on 
PyPI (I have no idea what is involved, but I presume it could be antomated.)


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com