Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
On 07/06/2016 05:11 AM, Steven D'Aprano wrote:
> On Tue, Jul 05, 2016 at 08:01:43PM +0200, Petr Viktorin wrote:
>
>> In the tkinter case, compiling for source is easy on a developer's
>> computer, but doing that on a headless server brings in devel files for
>> the entire graphical environment.
>> Are you saying Python on servers should have a way to do turtle
>> graphics, otherwise it's not Python?
>
> That's a really good question.
>
> I don't think we have an exact answer to "What counts as Python?". It's
> not like EMCAScript (Javascript) or C where there's a standard that
> defines the language and standard modules. We just have some de facto
> guidelines:
>
> - CPython is definitely Python;
> - Jython is surely Python, even if it lacks the byte-code of CPython and
> some things behave slightly differently;
> - MicroPython is probably Python, because nobody expects to be able to
> run Tkinter GUI apps on an embedded device with 256K or RAM;
>
> but it's hard to make that judgement except on a case-by-case basis.
>
> I think though that even if there's no documented line, most people
> recognise that there are "core" and "non-core" standard modules. dis and
> tkinter are non-core: if µPython leaves out tkinter, nobody will be
> surprised; if Jython leaves out dis, nobody will hold it against them;
> but if they leave out math or htmllib that's another story.
For MicroPython, I would definitely expect htmllib to be an optional
add-on – it's not useful for reading data off a thermometer saving it to
an SD card. But I guess that's getting too deep into specifics.
> So a headless server can probably leave out tkinter; but a desktop
> shouldn't.
Up till recently this wasn't possible to express in terms of RPM
dependencies. Now, it's on the ever-growing TODO list...
Another problem here is that you don't explicitly "install Python" on
Fedora: when you install the system, you get a minimal set of packages
to make everything work, and most of Python is part of that – but
tkinter is not. This is in contrast to python.org releases, where you
explicitly ask for (all of) Python.
Technically it would now be possible to have to install Python to use
it, but we run into another "batteries included" problem: Python (or,
"most-of-Python") is a pretty good battery for an OS.
Maybe a good short-term solution would be to make "import tkinter" raise
ImportError("Run `dnf install tkinter` to install the tkinter module")
if not found. This would prevent confusion while keeping the status quo.
I'll look into that.
> [...]
>>> The other extreme is Javascript/Node.js, where the "just use pip" (or
>>> npm in this case) philosophy has been taken to such extremes that one
>>> developer practically brought down the entire Node.js ecosystem by
>>> withdrawing an eleven line module, left-pad, in a fit of pique.
>>>
>>> Being open source, the damage was routed around quite quickly, but
>>> still, I think it's a good cautionary example of how a technological
>>> advance can transform a programming culture to the worse.
>>
>> I don't understand the analogy. Should the eleven-line module have been
>> in Node's stdlib? Outside of stdlib, people are doing this.
>
> The point is that Javascript/Node.js is so lacking in batteries that the
> community culture has gravitated to an extreme version of "just use
> pip". I'm not suggesting that you, or anyone else, has proposed that
> Python do the same, only that there's a balance to be found between the
> extremes of "everything in the Python ecosystem should be part of the
> standard installation" and "next to nothing should be part of the
> standard installation".
>
> The hard part is deciding where that balance should be :-)
I think the balance is where it needs to be for CPython, and it's also
where it needs to be for Fedora. The real hard part is acknowledging
that it needs to be in different places for different use cases, and
making sure work to support the different use cases is coordinated.
So, I guess I'm starting to form a concrete proposal:
1) Document what should happen when a stdlib module is not available.
This should be an ImportError informative error message, something along
the lines of 'This build of Python does not include SQLite support.' or
'MicroPython does not support turtle' or 'Use `sudo
your-package-manager` install tkinter` to install this module'.
2) Document leaf modules (or "clusters") that can be removed from the
stdlib, and their dependencies. Make no guarantees about cross-version
compatibility of this metadata.
3) Standardize a way to query which stdlib modules are present (without
side effects, i.e. trying an import doesn't count)
4) Adjust pip to ignore installed stdlib modules that are present, so
distributions can depend on "statistics" and not "statistics if
python_ver<3.4". (statistics is just an example, obviously this would
only work for modules added after the PEP). For missing stdlib modules,
pip should f
Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
On Wed, Jul 6, 2016 at 7:01 PM, Petr Viktorin wrote:
> Maybe a good short-term solution would be to make "import tkinter" raise
> ImportError("Run `dnf install tkinter` to install the tkinter module")
> if not found. This would prevent confusion while keeping the status quo.
> I'll look into that.
>
+1. There's precedent for it; Debian does this:
rosuav@sikorsky:~$ python
Python 2.7.11+ (default, Jun 2 2016, 19:34:15)
[GCC 5.3.1 20160528] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Tkinter
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 42, in
raise ImportError, str(msg) + ', please install the python-tk package'
ImportError: No module named _tkinter, please install the python-tk package
ChrisA
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
On 6 July 2016 at 10:01, Petr Viktorin wrote: > 4) Adjust pip to ignore installed stdlib modules that are present, so > distributions can depend on "statistics" and not "statistics if > python_ver<3.4". (statistics is just an example, obviously this would > only work for modules added after the PEP). For missing stdlib modules, > pip should fail with the message from 1). (Unless the "pip upgrade > asynciio" proposal goes through, in which case install the module if > it's upgradable). A couple of comments here. 1. Projects may still need to depend on "statistics from Python 3.6 or later, but the one in 3.5 isn't good enough". Consider for example unittest, where projects often need the backport unittest2 to get access to features that aren't in older versions. 2. This is easy enough to do if we make stdlib modules publish version metadata. But it does raise the question of what the version of a stdlib module is - probably Python version plus a micro version for interim updates. Also, I have a recollection of pip having problems with some stdlib modules that publish version data right now (wsgiref?) - that should be checked to make sure this approach would work. > Looks like it's time for a PEP. Probably - in principle, something like this proposal could be workable, it'll be a matter of thrashing out the details (which is something the PEP process is good at). Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
I think the wsgiref issue was that it wasn't in site-packages and so couldn't be removed or upgraded. Having the dist-info available and putting them in site-packages (or a new directory?) lets us handle querying/replacing/removing using the existing tools we use for distribution. Also, I think the version numbers do need to be independent of Python version in case nothing changes between releases. If you develop something using statistics==3.7, but statistics==3.6 is identical, how do you know you can put the lower constraint? Alternatively, if it's statistics==1.0 in both, people won't assume it has anything to do with the Python version. The tricky part here is when everything is in the one repo and everyone implicitly uses the latest version for development, you get the reproducibility issues Barry mentioned earlier. If they're separate and most people have pinned versions, that goes away. Top-posted from my Windows Phone -Original Message- From: "Paul Moore" Sent: 7/6/2016 2:32 To: "Petr Viktorin" Cc: "Python-Dev" Subject: Re: [Python-Dev] Breaking up the stdlib (Was: release cadence) On 6 July 2016 at 10:01, Petr Viktorin wrote: > 4) Adjust pip to ignore installed stdlib modules that are present, so > distributions can depend on "statistics" and not "statistics if > python_ver<3.4". (statistics is just an example, obviously this would > only work for modules added after the PEP). For missing stdlib modules, > pip should fail with the message from 1). (Unless the "pip upgrade > asynciio" proposal goes through, in which case install the module if > it's upgradable). A couple of comments here. 1. Projects may still need to depend on "statistics from Python 3.6 or later, but the one in 3.5 isn't good enough". Consider for example unittest, where projects often need the backport unittest2 to get access to features that aren't in older versions. 2. This is easy enough to do if we make stdlib modules publish version metadata. But it does raise the question of what the version of a stdlib module is - probably Python version plus a micro version for interim updates. Also, I have a recollection of pip having problems with some stdlib modules that publish version data right now (wsgiref?) - that should be checked to make sure this approach would work. > Looks like it's time for a PEP. Probably - in principle, something like this proposal could be workable, it'll be a matter of thrashing out the details (which is something the PEP process is good at). Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
On 6 July 2016 at 14:55, Steve Dower wrote: > I think the wsgiref issue was that it wasn't in site-packages and so > couldn't be removed or upgraded. Having the dist-info available and putting > them in site-packages (or a new directory?) lets us handle > querying/replacing/removing using the existing tools we use for > distribution. That's the one. Thanks for the reminder. So either we move the stdlib (or parts of it) into site-packages, or pip needs to learn to handle a versioned stdlib. Cool. > Also, I think the version numbers do need to be independent of Python > version in case nothing changes between releases. If you develop something > using statistics==3.7, but statistics==3.6 is identical, how do you know you > can put the lower constraint? Alternatively, if it's statistics==1.0 in > both, people won't assume it has anything to do with the Python version. This boils down to whether we want to present the stdlib as a unified object tied to the Python release, or a set of modules no different from those on PyPI, that happen to be shipped with Python. I prefer the former view (it matches better how I think of "batteries included") whereas it looks like you prefer the latter (but don't see that as being in conflict with "batteries included"). Debating this in the abstract is probably not productive, so let's wait for a concrete PEP to thrash out details like this. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Making stdlib modules optional for distributions (Was: Breaking up the stdlib (Was: release cadence))
I consider making stdlib modules "optional" like this to be completely separate
from making them individually versioned - can't quite tell whether you guys do
as well?
To everyone: please don't conflate these two discussions. The other is about
CPython workflow and this one is about community/user expectations (I have not
been proposing to remove stdlib modules at any point).
Cheers,
Steve
Top-posted from my Windows Phone
-Original Message-
From: "Petr Viktorin"
Sent: 7/6/2016 2:04
To: "Steven D'Aprano" ; "Python-Dev"
Subject: Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
On 07/06/2016 05:11 AM, Steven D'Aprano wrote:
> On Tue, Jul 05, 2016 at 08:01:43PM +0200, Petr Viktorin wrote:
>
>> In the tkinter case, compiling for source is easy on a developer's
>> computer, but doing that on a headless server brings in devel files for
>> the entire graphical environment.
>> Are you saying Python on servers should have a way to do turtle
>> graphics, otherwise it's not Python?
>
> That's a really good question.
>
> I don't think we have an exact answer to "What counts as Python?". It's
> not like EMCAScript (Javascript) or C where there's a standard that
> defines the language and standard modules. We just have some de facto
> guidelines:
>
> - CPython is definitely Python;
> - Jython is surely Python, even if it lacks the byte-code of CPython and
> some things behave slightly differently;
> - MicroPython is probably Python, because nobody expects to be able to
> run Tkinter GUI apps on an embedded device with 256K or RAM;
>
> but it's hard to make that judgement except on a case-by-case basis.
>
> I think though that even if there's no documented line, most people
> recognise that there are "core" and "non-core" standard modules. dis and
> tkinter are non-core: if µPython leaves out tkinter, nobody will be
> surprised; if Jython leaves out dis, nobody will hold it against them;
> but if they leave out math or htmllib that's another story.
For MicroPython, I would definitely expect htmllib to be an optional
add-on – it's not useful for reading data off a thermometer saving it to
an SD card. But I guess that's getting too deep into specifics.
> So a headless server can probably leave out tkinter; but a desktop
> shouldn't.
Up till recently this wasn't possible to express in terms of RPM
dependencies. Now, it's on the ever-growing TODO list...
Another problem here is that you don't explicitly "install Python" on
Fedora: when you install the system, you get a minimal set of packages
to make everything work, and most of Python is part of that – but
tkinter is not. This is in contrast to python.org releases, where you
explicitly ask for (all of) Python.
Technically it would now be possible to have to install Python to use
it, but we run into another "batteries included" problem: Python (or,
"most-of-Python") is a pretty good battery for an OS.
Maybe a good short-term solution would be to make "import tkinter" raise
ImportError("Run `dnf install tkinter` to install the tkinter module")
if not found. This would prevent confusion while keeping the status quo.
I'll look into that.
> [...]
>>> The other extreme is Javascript/Node.js, where the "just use pip" (or
>>> npm in this case) philosophy has been taken to such extremes that one
>>> developer practically brought down the entire Node.js ecosystem by
>>> withdrawing an eleven line module, left-pad, in a fit of pique.
>>>
>>> Being open source, the damage was routed around quite quickly, but
>>> still, I think it's a good cautionary example of how a technological
>>> advance can transform a programming culture to the worse.
>>
>> I don't understand the analogy. Should the eleven-line module have been
>> in Node's stdlib? Outside of stdlib, people are doing this.
>
> The point is that Javascript/Node.js is so lacking in batteries that the
> community culture has gravitated to an extreme version of "just use
> pip". I'm not suggesting that you, or anyone else, has proposed that
> Python do the same, only that there's a balance to be found between the
> extremes of "everything in the Python ecosystem should be part of the
> standard installation" and "next to nothing should be part of the
> standard installation".
>
> The hard part is deciding where that balance should be :-)
I think the balance is where it needs to be for CPython, and it's also
where it needs to be for Fedora. The real hard part is acknowledging
that it needs to be in different places for different use cases, and
making sure work to support the different use cases is coordinated.
So, I guess I'm starting to form a concrete proposal:
1) Document what should happen when a stdlib module is not available.
This should be an ImportError informative error message, something along
the lines of 'This build of Python does not include SQLite support.' or
'MicroPython does not support turtle' or 'Use `sudo
your-package-manager` install tk
Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
Thrashing out details should go on the workflow SIG, and I guess I'm the obvious candidate to push it asking. But given my own time constraints right now, I'm not going to dive into details if the high level concept (stdlib packages can be individually updated by end users apart from a full CPython release) is at issue. Once there seems to be general agreement that this is a worthy goal, I'll see if I can put down details for how I would implement it. (And go join the core-workflow list, I guess :) ) Top-posted from my Windows Phone -Original Message- From: "Paul Moore" Sent: 7/6/2016 7:10 To: "Steve Dower" Cc: "Petr Viktorin" ; "Python-Dev" Subject: Re: [Python-Dev] Breaking up the stdlib (Was: release cadence) On 6 July 2016 at 14:55, Steve Dower wrote: > I think the wsgiref issue was that it wasn't in site-packages and so > couldn't be removed or upgraded. Having the dist-info available and putting > them in site-packages (or a new directory?) lets us handle > querying/replacing/removing using the existing tools we use for > distribution. That's the one. Thanks for the reminder. So either we move the stdlib (or parts of it) into site-packages, or pip needs to learn to handle a versioned stdlib. Cool. > Also, I think the version numbers do need to be independent of Python > version in case nothing changes between releases. If you develop something > using statistics==3.7, but statistics==3.6 is identical, how do you know you > can put the lower constraint? Alternatively, if it's statistics==1.0 in > both, people won't assume it has anything to do with the Python version. This boils down to whether we want to present the stdlib as a unified object tied to the Python release, or a set of modules no different from those on PyPI, that happen to be shipped with Python. I prefer the former view (it matches better how I think of "batteries included") whereas it looks like you prefer the latter (but don't see that as being in conflict with "batteries included"). Debating this in the abstract is probably not productive, so let's wait for a concrete PEP to thrash out details like this. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)
On 06Jul2016 0753, Steve Dower wrote: Thrashing out details should go on the workflow SIG, and I guess I'm the obvious candidate to push it asking. But given my own time constraints right now, I'm not going to dive into details if the high level concept (stdlib packages can be individually updated by end users apart from a full CPython release) is at issue. Once there seems to be general agreement that this is a worthy goal, I'll see if I can put down details for how I would implement it. (And go join the core-workflow list, I guess :) ) Rather than wait for general agreement, since this thread is probably widely muted already, I'll put a PEP together to clearly set out what I'm envisioning here in a form we can directly discuss. But time is precious, so don't expect it this week :) (Also, on Brett's advice, this belongs on python-dev and not core-workflow right now.) Cheers, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Making stdlib modules optional for distributions (Was: Breaking up the stdlib (Was: release cadence))
On 7 July 2016 at 00:27, Steve Dower wrote: > I consider making stdlib modules "optional" like this to be completely > separate from making them individually versioned - can't quite tell whether > you guys do as well? The point of overlap I see is that if the stdlib starts putting some selected modules into site-packages (so "pip install --upgrade " works without any changes to pip or equivalent tools), then that also solves the "How to explicitly declare dependencies on particular pieces of the standard library" problem: you use the same mechanisms we already use to declare dependencies on 3rd party packages distributed via a packaging server. I really like the idea of those independently versioned subcomponents of the standard library only being special in the way they're developed (i.e. as part of the standard library) and the way they're published (i.e. included by default with the python.org binary installers and source tarballs), rather than also being special in the way they're managed at runtime. Versioning selected stdlib subsets also has potential to help delineate clear expectations for redistributors: Are you leaving a particular subset out of your default install? Then you should propose that it become an independently versioned subset of the stdlib. Is a given subset already independently versioned in the stdlib? Then it may be OK to leave it out of your default install and document that you've done so. > To everyone: please don't conflate these two discussions. The other is about > CPython workflow and this one is about community/user expectations (I have > not been proposing to remove stdlib modules at any point). While I agree they're separate discussions, the workflow management one has the potential to *also* improve the user experience in cases where redistributors are already separating out pieces of the stdlib to make them optional. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
