[Python-Dev] Breaking up the stdlib (Was: release cadence)

[Petr Viktorin]

On 07/04/2016 12:19 AM, Steve Dower wrote:
> My thinking on this issue was that some/most packages from the stdlib
> would move into site-packages. Certainly I'd expect asyncio to be in
> this category, and probably typing. Even going as far as email and
> urllib would potentially be beneficial (to those packages, is my thinking).
> 
> Obviously not every single module can do this, but there are plenty that
> aren't low-level dependencies for other modules that could. Depending on
> particular versions of these then becomes a case of adding normal
> package version constraints - we could even bundle version information
> for non-updateable packages so that installs fail on incompatible Python
> versions.
> 
> The "Uber repository" could be a requirements.txt that pulls down wheels
> for the selected stable versions of each package so that we still
> distribute all the same code with the same stability, but users have
> much more ability to patch their own stdlib after install.
> 
> (FWIW, we use a system similar to this at Microsoft for building Visual
> Studio, so I can vouch that it works on much more complicated software
> than Python.)

While we're on the subject, I'd like to offer another point for
consideration: not all implementations of Python can provide the full
stdlib, and not everyone wants the full stdlib.

For MicroPython, most of Python's batteries are too heavy. Tkinter on
Android is probably not useful enough for people to port it. Weakref
can't be emulated nicely in Javascript.
If packages had a way to opt-out of needing the whole standard library,
and instead specify the stdlib subset they need, answering questions
like "will this run on my phone?" and "what piece of the stdlib do we
want to port next?" would be easier.

Both Debian and Fedora package some parts of the stdlib separately
(tkinter, venv, tests), and have opt-in subsets of the stdlib for
minimal systems (python-minimal, system-python). Tools like pyinstaller
run magic heuristics to determine what parts of stdlib can be left out.
It would help these projects if the "not all of stdlib is installed"
case was handled more systematically at the CPython or distutils level.

As I said on the Language summit, this is just talk; I don't currently
have the resources to drive this effort. But if anyone is thinking of
splitting the stdlib, please keep these points in mind as well. I think
that, at least, if "pip install -U asyncio" becomes possible, "pip
uninstall --yes-i-know-what-im-doing asyncio" should be possible as well.



> From: Paul Moore 
> Sent: ‎7/‎3/‎2016 14:23
> To: Brett Cannon 
> Cc: Guido van Rossum ; Nick Coghlan
> ; Python-Dev ;
> Steve Dower 
> Subject: Re: [Python-Dev] release cadence (was: Request for CPython
> 3.5.3 release)
> 
> On 3 July 2016 at 22:04, Brett Cannon  wrote:
>> This last bit is what I would advocate if we broke the stdlib out
> unless an
>> emergency patch release is warranted for a specific module (e.g. like
>> asyncio that started this discussion). Obviously backporting is its own
>> thing.
> 
> It's also worth noting that pip has no mechanism for installing an
> updated stdlib module, as everything goes into site-packages, and the
> stdlib takes precedence over site-packages unless you get into
> sys.path hacking abominations like setuptools uses (or at least used
> to use, I don't know if it still does). So as things stand,
> independent patch releases of stdlib modules would need to be manually
> copied into place.
> 
> Allowing users to override the stdlib opens up a different can of
> worms - not necessarily one that we couldn't resolve, but IIRC, it was
> always a deliberate policy that overriding the stdlib wasn't possible
> (that's why backports have names like unittest2...)
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Chris Angelico]

On Tue, Jul 5, 2016 at 5:53 PM, Petr Viktorin  wrote:
> If packages had a way to opt-out of needing the whole standard library,
> and instead specify the stdlib subset they need, answering questions
> like "will this run on my phone?" and "what piece of the stdlib do we
> want to port next?" would be easier.

On the flip side, answering questions like "what version of Python do
people need to run my program" become harder to answer, particularly
if you have third-party dependencies. (The latest version of numpy
might decide that it's going to 'import statistics', for instance.)
One of the arguments against splitting the stdlib was that corporate
approval for software is often hard to obtain, and it's much easier to
say "I need approval to use Python, exactly as distributed by
python.org" than "I need approval to use Python-core plus these five
Python-stdlib sections".

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Petr Viktorin]

On 07/05/2016 10:05 AM, Chris Angelico wrote:
> On Tue, Jul 5, 2016 at 5:53 PM, Petr Viktorin  wrote:
>> If packages had a way to opt-out of needing the whole standard library,
>> and instead specify the stdlib subset they need, answering questions
>> like "will this run on my phone?" and "what piece of the stdlib do we
>> want to port next?" would be easier.
> 
> On the flip side, answering questions like "what version of Python do
> people need to run my program" become harder to answer, particularly
> if you have third-party dependencies. (The latest version of numpy
> might decide that it's going to 'import statistics', for instance.)

That question is already hard to answer. How do you tell if a library
works on Micropython? Or Python for Android?

I'm not arguing to change the default; if the next version of numpy
doesn't do anything, nothing should change. However, under the status
quo, "Python 3.4" means "CPython 3.4 with the full stdlib, otherwise all
bets are off", and there's no good way to opt in to more granularity.


> One of the arguments against splitting the stdlib was that corporate
> approval for software is often hard to obtain, and it's much easier to
> say "I need approval to use Python, exactly as distributed by
> python.org" than "I need approval to use Python-core plus these five
> Python-stdlib sections".

I'm not arguing against "Python, exactly as distributed by python.org"
not including all of stdlib.
I would like making stripped-down variants of CPython easier, and to
make it possible to opt-in to use CPython without all of stdlib, so that
major problems with stdlib availablility in other Python implementations
can be caught early.
Basically, instead of projects getting commits like "Add metadata for
one flavor of Android packaging tool", I'd like to see "Add common
metadata for Android, IPhone, PyInstaller, and minimal Linux, and make
sure the CPython-based CI smoke-tests that metadata".

Also, I believe corporate approval for python.org's Python is a bit of a
red herring – usually you'd get approval for Python distributed by
Continuum or Red Hat or Canonical or some such. As a Red Hat employee, I
can say that what I'm suggesting won't make me suffer, and I see no
reason it would hurt the others either.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Victor Stinner]

Hi,

asyncio is a good example because it wants to evolve faster than the
whole "CPython package". Each minor version of CPython adds news
features in asyncio. It is not always easy to document these changes.
Moreover, the behaviour of some functions also changed in minor
versions.

asyncio doesn't respect the trend of semantic versions
http://semver.org/ The major version should change if the behaviour of
an existing function changes.


2016-07-05 10:05 GMT+02:00 Chris Angelico :
> On the flip side, answering questions like "what version of Python do
> people need to run my program" become harder to answer, particularly
> if you have third-party dependencies. (The latest version of numpy
> might decide that it's going to 'import statistics', for instance.)

Recently, I wrote a "perf" module and I wanted to use the statistics
module. I was surprised to see that PyPI has a package called
"statistics" which just works on Python 2.7. In practice, I can use
statistics on Python 2.7, 3.4 and even 3.2 (but I didn't try, this
version is too old). It's a matter of describing correctly
dependencies.

pip supports a requirements.txt file which is a nice may to declare
dependency. You can:

* specify the minimum library version
* make some library specific to some operation systems
* skip dependencies on some Python versions -- very helpful for
libraries parts of Python 3 stdlib (like statistics)

=> see Environment markers for conditions on dependencies

For perf, I'm using this setup() option in setup.py:

'install_requires': ["statistics; python_version < '3.4'", "six"],


> One of the arguments against splitting the stdlib was that corporate
> approval for software is often hard to obtain, and it's much easier to
> say "I need approval to use Python, exactly as distributed by
> python.org" than "I need approval to use Python-core plus these five
> Python-stdlib sections".

*If* someone wants to split the stdlib into smaller parts and/or move
it out of CPython, you should already start to write a PEP. Or you
will have to reply to the same questions over and over ;-)

Is there a volunteer to write such PEP?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for CPython 3.5.3 release

[Barry Warsaw]

On Jul 03, 2016, at 01:17 AM, Ludovic Gasc wrote:

>If 3.5.2.1 or 3.5.3 are impossible to release before december, what are the
>alternative solutions for AsyncIO users ?
>1. Use 3.5.1 and hope that Linux distributions won't use 3.5.2 ?

Matthias just uploaded a 3.5.2-2 to Debian unstable, which will also soon make
its way to Ubuntu 16.10:

https://launchpad.net/ubuntu/+source/python3.5/3.5.2-2

Ubuntu 16.04 LTS currently still has 3.5.1.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

> This is an area of exceeding subtlety (and also not very well
> documented/specified, probably). I'd worry that changing anything here
> might break some code. When a metaclass overrides neither __init__ nor
> __new__, keyword args will not work because type.__init__ forbids
> them. However when a metaclass overrides them and calls them using
> super(), it's quite possible that someone ended up calling
> super().__init__() with three positional args but super().__new__()
> with keyword args, since the call sites are distinct (in the overrides
> for __init__ and __new__ respectively).
>
> What's your argument for changing this, apart from a desire for more 
> regularity?

The implementation gets much simpler if __new__ doesn't take keyword
arguments. It's simply that if it does, I have to filter out __new__'s
three arguments.
That's easily done in Python, unfortunately not so much in C.

So we have two options: either type.__new__ is limited to accepting positional
arguments only, possibly breaking some code, but which could be changed
easily. This leads to a pretty simple implementation: pass over
keyword arguments
to __init_subclass__, that's it.

The other option is: filter out name, bases and dict from the keyword arguments
If people think that backwards compatibility is that important, I
could do that. But
that just leaves quite some code in places where there is already a lot of
complicated code.

Nick proposed a compromise, just don't filter for name, bases and dict, and
pass them over to __init_subclass__. Then the default implementation of
__init_subclass__ must support those three keyword arguments and do
nothing with them.

I'm fine with all three solutions, although I have a preference for the first.
I think passing keyword arguments to type.__new__ is already really rare
and if it does exist, it's super easy to fix.

> I'm confused. In the above example it would seem that the keyword args
> {'a': 1, 'b': 2} are passed right on to super9).__init_subclass__().
> Do you mean that it ignores all keyword args? Or that it has no
> positional args? (Both of which would be consistent with the example.)

The example is just wrong. I'll fix it.

> Can you state exactly at which point during class initialization
> __init_class__() is called? (Surely by now, having implemented it, you
> know exactly where. :-)

Further down in the PEP I give the exact

> [This is as far as I got reviewing when the weekend activities
> interrupted me. In the light of ongoing discussion I'm posting this
> now -- I'll continue later.]

I hope you had a good weekend not thinking too much about
metaclasses...

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Webmaster] Unsafe listing by Norton's "File Insight"

[Steve Dower]

On 04Jul2016 2241, Steve Holden wrote:

Hi Peter,

While the humble webmasters can do little about this it's possible the
developers can, so I am forwarding your email to their mailing list.

regards
 Steve

Steve Holden

On Tue, Jul 5, 2016 at 3:30 AM, Peter via Webmaster
mailto:webmas...@python.org>> wrote:

Hi
I'm a heavy user of Python on Windows, am a Basic PSF member and
have contributed to core Python.
The Python 2.7.12 Windows installer download is being marked as
untrusted by Norton Internet Security. I've been on chat with
Symantec, and they've said that I can't do anything about that
rating, but that the site owner can.
I've been pointed to:
https://safeweb.norton.com/help/site_owners
Interestingly, the 3.5.2 download is flagged as safe.
Hoping to get more Python out to users!
Thanks
Peter


Peter, can you provide the exact URL that safeweb is complaining about? 
I tried a few at https://safeweb.norton.com/ and they all showed up as 
clean.


Also please clarify whether this is what you mean. It's not entirely 
clear whether the download is being scanned or the reputation of the URL 
is in question.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Steven D'Aprano]

On Tue, Jul 05, 2016 at 09:53:24AM +0200, Petr Viktorin wrote:

> While we're on the subject, I'd like to offer another point for
> consideration: not all implementations of Python can provide the full
> stdlib, and not everyone wants the full stdlib.
> 
> For MicroPython, most of Python's batteries are too heavy. Tkinter on
> Android is probably not useful enough for people to port it. Weakref
> can't be emulated nicely in Javascript.
> If packages had a way to opt-out of needing the whole standard library,
> and instead specify the stdlib subset they need, answering questions
> like "will this run on my phone?" and "what piece of the stdlib do we
> want to port next?" would be easier.

I don't know that they will be easier. That seems pretty counter- 
intuitive to me. At the moment, answering these questions are really 
easy if you use nothing but the std lib: the answer is, if you can 
install Python, it will work. As soon as you start using non-stdlib 
modules, the question becomes:

- have you installed Python? have you installed module X? and module Y? 
  and module Z? do they match the version of the interpreter? where 
  did you get them from? are you missing dependencies?

I can't tell you how much trouble I've had trying to get tkinter working 
on some Fedora systems because they split tkinter into a separate 
package. Sure, if I had *known* that it was split into a separate 
package, then just running `yum install packagename` would (probably!) 
have worked, but how was I supposed to know? It's not documented 
anywhere that I could find. I ended up avoiding the Fedora packages and 
installing from source.

I think there comes a time in every successful organisation that they 
risk losing sight of what made them successful in the first place. (And, 
yes, I'm aware that the *other* way that successful organisations lose 
their way is by failing to change with the times.)

Yes, we're all probably sick and tired of hearing all the Chicken Little 
scare stories about how the GIL is killing Python, how everyone is 
abandoning Python for Ruby/Javascript/Go/Swift, how Python 3 is killing 
Python, etc. But sometimes the sky does fall. For many people, Python's 
single biggest advantage until now has been "batteries included", and I 
think that changing that is risky and shouldn't be done lightly.

It's easy to say "just use pip", but if you've ever been stuck behind a 
corporate firewall where pip doesn't work, or where dowloading and 
installing software is a firing offence, then you might think 
differently. If you've had to teach a room full of 14 year olds, and you 
spend the entire lesson just helping them to install one library, you 
might have a different view.

The other extreme is Javascript/Node.js, where the "just use pip" (or 
npm in this case) philosophy has been taken to such extremes that one 
developer practically brought down the entire Node.js ecosystem by 
withdrawing an eleven line module, left-pad, in a fit of pique.

Being open source, the damage was routed around quite quickly, but 
still, I think it's a good cautionary example of how a technological 
advance can transform a programming culture to the worse.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Paul Moore]

On 5 July 2016 at 18:02, Steven D'Aprano  wrote:
> Yes, we're all probably sick and tired of hearing all the Chicken Little
> scare stories about how the GIL is killing Python, how everyone is
> abandoning Python for Ruby/Javascript/Go/Swift, how Python 3 is killing
> Python, etc. But sometimes the sky does fall. For many people, Python's
> single biggest advantage until now has been "batteries included", and I
> think that changing that is risky and shouldn't be done lightly.

+1

To be fair, I don't think anyone is looking at this "lightly", but I
do think it's easy to underestimate the value of "batteries included",
and the people it's *most* useful for are precisely the people who
aren't involved in any of the Python mailing lists. They just want to
get on with things, and "it came with the language" is a *huge*
selling point.

Internal changes in how we manage the stdlib modules are fine. But
changing what the end user sees as "python" is a much bigger deal.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Steve Dower]

On 05Jul2016 1021, Paul Moore wrote:

On 5 July 2016 at 18:02, Steven D'Aprano  wrote:

Yes, we're all probably sick and tired of hearing all the Chicken Little
scare stories about how the GIL is killing Python, how everyone is
abandoning Python for Ruby/Javascript/Go/Swift, how Python 3 is killing
Python, etc. But sometimes the sky does fall. For many people, Python's
single biggest advantage until now has been "batteries included", and I
think that changing that is risky and shouldn't be done lightly.


+1

To be fair, I don't think anyone is looking at this "lightly", but I
do think it's easy to underestimate the value of "batteries included",
and the people it's *most* useful for are precisely the people who
aren't involved in any of the Python mailing lists. They just want to
get on with things, and "it came with the language" is a *huge*
selling point.

Internal changes in how we manage the stdlib modules are fine. But
changing what the end user sees as "python" is a much bigger deal.


Also +1 on this - a default install of Python should continue to include 
everything it currently does.


My interest in changing anything at all is to provide options for 
end-users/distributors to either reduce the footprint (which they 
already do), to more quickly update specific modules, and perhaps 
long-term to make user's code be less tied to a particular Python 
version (instead being tied to, for example, a specific asyncio version 
that can be brought into a range of supported Python versions).


Batteries included is a big deal.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Barry Warsaw]

On Jul 05, 2016, at 11:19 AM, Victor Stinner wrote:

>pip supports a requirements.txt file which is a nice may to declare
>dependency. You can:
>
>* specify the minimum library version
>* make some library specific to some operation systems
>* skip dependencies on some Python versions -- very helpful for
>libraries parts of Python 3 stdlib (like statistics)

Interestingly enough, I'm working on a project where we *have* to use packages
from the Ubuntu archive, even if there are different (or differently fixed)
versions on PyPI.  I don't think there's a way to map a requirements.txt into
distro package versions and do the install from the distro package manager,
but that might be useful.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence

[Barry Warsaw]

On Jul 03, 2016, at 04:21 PM, Steve Dower wrote:

>A rough count of how I'd break up the current 3.5 Lib folder (which I
>happened to have handy) suggests no more than 50 repos.

A concern with a highly split stdlib is local testing.  I'm not worried about
pull request testing, or after-the-fact buildbot testing since I'd have to
assume that we'd make sure the fully integrated sumo package was tested in
both environments.

But what about local testing?  Let's say you change something in one module
that causes a regression in a different module in a different repo.  If you've
only got a small subset checked out, you might never notice that before you
PR'd your change.  And then once the test fails, how easy will it be for you
to recreate the tested environment locally so that you could debug your
regression?

I'm sure it's doable, but let's not lose sight of that if this path is taken.

(Personally, I'm +0 on splitting out the stdlib and -1 on micro-splitting it.)

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence

[Steve Dower]

On 05Jul2016 1028, Barry Warsaw wrote:

On Jul 03, 2016, at 04:21 PM, Steve Dower wrote:


A rough count of how I'd break up the current 3.5 Lib folder (which I
happened to have handy) suggests no more than 50 repos.


A concern with a highly split stdlib is local testing.  I'm not worried about
pull request testing, or after-the-fact buildbot testing since I'd have to
assume that we'd make sure the fully integrated sumo package was tested in
both environments.

But what about local testing?  Let's say you change something in one module
that causes a regression in a different module in a different repo.  If you've
only got a small subset checked out, you might never notice that before you
PR'd your change.  And then once the test fails, how easy will it be for you
to recreate the tested environment locally so that you could debug your
regression?

I'm sure it's doable, but let's not lose sight of that if this path is taken.


My hope is that it would be essentially a "pip freeze"/"pip install -r 
..." (or equivalent with whatever tool is used/created for managing the 
stdlib). Perhaps using VCS URIs rather than version numbers?


That is, the test run would dump a list of exactly which stdlib versions 
it's using, so that when you review the results it is possible to 
recreate it.


But the point is well taken. I'm very hesitant about splitting out 
packages that are common dependencies of other parts of the stdlib, but 
there are plenty of leaf nodes in there too. Creating a complex 
dependency graph would be a disaster.


Cheers,
Steve


(Personally, I'm +0 on splitting out the stdlib and -1 on micro-splitting it.)

Cheers,
-Barry


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

[Barry Warsaw]

On Jul 04, 2016, at 10:31 AM, Nick Coghlan wrote:

>While we liked the "consistent calendar cadence that is some multiple
>of 6 months" idea, several of us thought 12 months was way too short
>as it makes for too many entries in third party support matrices.

18 months for a major release cadence still seems right to me.  Downstreams
and third-parties often have to go through *a lot* of work to ensure
compatibility, and try as we might, every Python release breaks *something*.
Major version releases trigger a huge cascade of other work for lots of other
people, and I don't think shortening that would be for the overall community
good.  It just feels like we'd always be playing catch up.

Different downstreams have different cadences.  I can only speak for Debian,
which has a release-when-ready policy, and Ubuntu, which has strictly timed
releases.  When the Python release aligns nicely with Ubuntu's LTS releases,
we can usually manage the transition fairly well because we can allocate
resource way ahead of time.  (I'm less concerned about Ubuntu's mid-LTS 6
month releases.)

For example, 3.6 final will come out in December 2016, so it'll be past our
current 16.10 Ubuntu release.  We've pretty much decided to carry Python 3.5
through until 17.04, and that'll give us a good year to make 18.04 LTS have a
solid Python 3.6 ecosystem.

Projecting ahead, it probably means 3.7 in mid-2018, which is after the Ubuntu
18.04 LTS release, so we'll only do one major transition before the next LTS.
>From my perspective, that feels about right.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence

[Barry Warsaw]

On Jul 05, 2016, at 10:38 AM, Steve Dower wrote:

>My hope is that it would be essentially a "pip freeze"/"pip install -r ..."
>(or equivalent with whatever tool is used/created for managing the
>stdlib). Perhaps using VCS URIs rather than version numbers?
>
>That is, the test run would dump a list of exactly which stdlib versions it's
>using, so that when you review the results it is possible to recreate it.

I think you'd have to have vcs checkouts though, because you will often need
to fix or change something in one of those other library pieces.  The other
complication of course is that now you'll have two dependent PRs with reviews
in two different repos.

>But the point is well taken. I'm very hesitant about splitting out packages
>that are common dependencies of other parts of the stdlib, but there are
>plenty of leaf nodes in there too. Creating a complex dependency graph would
>be a disaster.

Yeah.  

Cheers,
-Barry


pgpMeHsirBIoe.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Petr Viktorin]

(sorry if you get this twice, I dropped python-dev by mistake)

On 07/05/2016 07:02 PM, Steven D'Aprano wrote:
> On Tue, Jul 05, 2016 at 09:53:24AM +0200, Petr Viktorin wrote:
> 
>> While we're on the subject, I'd like to offer another point for
>> consideration: not all implementations of Python can provide the full
>> stdlib, and not everyone wants the full stdlib.
>>
>> For MicroPython, most of Python's batteries are too heavy. Tkinter on
>> Android is probably not useful enough for people to port it. Weakref
>> can't be emulated nicely in Javascript.
>> If packages had a way to opt-out of needing the whole standard library,
>> and instead specify the stdlib subset they need, answering questions
>> like "will this run on my phone?" and "what piece of the stdlib do we
>> want to port next?" would be easier.
> 
> I don't know that they will be easier. That seems pretty counter- 
> intuitive to me. At the moment, answering these questions are really 
> easy if you use nothing but the std lib: the answer is, if you can 
> install Python, it will work. As soon as you start using non-stdlib 
> modules, the question becomes:
> 
> - have you installed Python? have you installed module X? and module Y? 
>   and module Z? do they match the version of the interpreter? where 
>   did you get them from? are you missing dependencies?
> 
> I can't tell you how much trouble I've had trying to get tkinter working 
> on some Fedora systems because they split tkinter into a separate 
> package. Sure, if I had *known* that it was split into a separate 
> package, then just running `yum install packagename` would (probably!) 
> have worked, but how was I supposed to know? It's not documented 
> anywhere that I could find. I ended up avoiding the Fedora packages and 
> installing from source.

Ah, but successfully installing from source doesn't always give you the
full stdlib either:

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_sqlite3
To find the necessary bits, look in setup.py in detect_modules() for the
module's name.

I have missed that message before, and wondered pretty hard why the
module wasn't there.

In the tkinter case, compiling for source is easy on a developer's
computer, but doing that on a headless server brings in devel files for
the entire graphical environment.
Are you saying Python on servers should have a way to do turtle
graphics, otherwise it's not Python?

> I think there comes a time in every successful organisation that they 
> risk losing sight of what made them successful in the first place. (And, 
> yes, I'm aware that the *other* way that successful organisations lose 
> their way is by failing to change with the times.)
> 
> Yes, we're all probably sick and tired of hearing all the Chicken Little 
> scare stories about how the GIL is killing Python, how everyone is 
> abandoning Python for Ruby/Javascript/Go/Swift, how Python 3 is killing 
> Python, etc. But sometimes the sky does fall. For many people, Python's 
> single biggest advantage until now has been "batteries included", and I 
> think that changing that is risky and shouldn't be done lightly.
> 
> It's easy to say "just use pip", but if you've ever been stuck behind a 
> corporate firewall where pip doesn't work, or where dowloading and 
> installing software is a firing offence, then you might think 
> differently. If you've had to teach a room full of 14 year olds, and you 
> spend the entire lesson just helping them to install one library, you 
> might have a different view.

That is why I'm *not* arguing for shipping an incomplete stdlib in
official Python releases. I fully agree that including batteries is
great – I'm just not a fan of welding the battery to the product.
There are people who want to cut out what they don't need – to build
self-contained applications (pyinstaller, Python for Android), or to
eliminate unnecessary dependencies (python3-tkinter). And they will do
it with CPython's blessing or without.
I don't think Python can move to the mobile world of self-contained apps
without this problem being solved, one way or another.
It would be much better for the ecosystem if CPython acknowledges this
and sets some rules (like "here's how you can do it, but don't call the
result an unqualified Python").

> The other extreme is Javascript/Node.js, where the "just use pip" (or 
> npm in this case) philosophy has been taken to such extremes that one 
> developer practically brought down the entire Node.js ecosystem by 
> withdrawing an eleven line module, left-pad, in a fit of pique.
> 
> Being open source, the damage was routed around quite quickly, but 
> still, I think it's a good cautionary example of how a technological 
> advance can transform a programming culture to the worse.

I don't understand the analogy. Should the eleven-line module have been
in Node's stdlib? Outside of stdlib, people are doing this.

___
P

Re: [Python-Dev] release cadence

[Ethan Furman]

On 07/05/2016 10:44 AM, Barry Warsaw wrote:

On Jul 04, 2016, at 10:31 AM, Nick Coghlan wrote:



While we liked the "consistent calendar cadence that is some multiple
of 6 months" idea, several of us thought 12 months was way too short
as it makes for too many entries in third party support matrices.


18 months for a major release cadence still seems right to me.  Downstreams
and third-parties often have to go through *a lot* of work to ensure
compatibility, and try as we might, every Python release breaks *something*.
Major version releases trigger a huge cascade of other work for lots of other
people, and I don't think shortening that would be for the overall community
good.  It just feels like we'd always be playing catch up.


+1 from me as well.  Rapid major releases are just a huge headache.  The 
nice thing about a .6 or .7 minor release is that we get closer to no 
bugs with each one.


--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

[Brett Cannon]

On Tue, 5 Jul 2016 at 10:45 Barry Warsaw  wrote:

> On Jul 04, 2016, at 10:31 AM, Nick Coghlan wrote:
>
> >While we liked the "consistent calendar cadence that is some multiple
> >of 6 months" idea, several of us thought 12 months was way too short
> >as it makes for too many entries in third party support matrices.
>
> 18 months for a major release cadence still seems right to me.  Downstreams
> and third-parties often have to go through *a lot* of work to ensure
> compatibility, and try as we might, every Python release breaks
> *something*.
> Major version releases trigger a huge cascade of other work for lots of
> other
> people, and I don't think shortening that would be for the overall
> community
> good.  It just feels like we'd always be playing catch up.
>

Sticking w/ 18 months is also fine, but then I would like to discuss
choosing what months we try to release to get into a date-based release
cadence so we know that every e.g. December and June are when releases
typically happen thanks to our 6 month bug-fix release cadence. This has
the nice benefit of all of us being generally aware of when a bug-fix
release is coming up instead of having to check the PEP or go through our
mail archive to find out what month a bug-fix is going to get cut (and also
something the community to basically count on).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Paul Moore]

On 5 July 2016 at 19:01, Petr Viktorin  wrote:
> There are people who want to cut out what they don't need – to build
> self-contained applications (pyinstaller, Python for Android), or to
> eliminate unnecessary dependencies (python3-tkinter). And they will do
> it with CPython's blessing or without.
[...]
> It would be much better for the ecosystem if CPython acknowledges this
> and sets some rules (like "here's how you can do it, but don't call the
> result an unqualified Python").

That doesn't sound unreasonable in principle. As a baseline, I guess
the current policy is essentially:

"""
You can build your own Python installation with whatever parts of the
stdlib omitted that you please. However, if you do this, you accept
responsibility for any consequences, in terms of 3rd-party modules not
working, or even stdlib breakage (for example, we don't guarantee that
parts of the stdlib may not rely on other parts).
"""

That's pretty simple, both to state and to adhere to. And it's
basically the current reality. What I'm not clear about is what
*additional* guarantees people want to make, and how we'd make them.
First of all, Python's packaging ecosystem has no way to express "this
package won't work if pathlib isn't present in your stdlib". It seems
to me that without something like that, it's pretty hard to support
anything better than the current position with regard to 3rd party
modules. Documenting stdlib inter-dependencies may be possible, but I
wouldn't like to make "X doesn't depend on Y" a guarantee that's
subject to backward compatibility rules.

Maybe the suggestion is to provide better tools for people wanting to
*build* such stripped down versions? That might be a possibility, I
guess. I don't know much about how people build their own copies of
Python to be able to comment.

So I guess my question is, what is the actual proposal here? People
seem to have concerns over things that aren't actually being proposed
- but without knowing what *is* being proposed, it's hard to avoid
that.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Brett Cannon]

On Tue, 5 Jul 2016 at 13:02 Paul Moore  wrote:

> On 5 July 2016 at 19:01, Petr Viktorin  wrote:
> > There are people who want to cut out what they don't need – to build
> > self-contained applications (pyinstaller, Python for Android), or to
> > eliminate unnecessary dependencies (python3-tkinter). And they will do
> > it with CPython's blessing or without.
> [...]
> > It would be much better for the ecosystem if CPython acknowledges this
> > and sets some rules (like "here's how you can do it, but don't call the
> > result an unqualified Python").
>
> That doesn't sound unreasonable in principle. As a baseline, I guess
> the current policy is essentially:
>
> """
> You can build your own Python installation with whatever parts of the
> stdlib omitted that you please. However, if you do this, you accept
> responsibility for any consequences, in terms of 3rd-party modules not
> working, or even stdlib breakage (for example, we don't guarantee that
> parts of the stdlib may not rely on other parts).
> """
>
> That's pretty simple, both to state and to adhere to. And it's
> basically the current reality. What I'm not clear about is what
> *additional* guarantees people want to make, and how we'd make them.
> First of all, Python's packaging ecosystem has no way to express "this
> package won't work if pathlib isn't present in your stdlib". It seems
> to me that without something like that, it's pretty hard to support
> anything better than the current position with regard to 3rd party
> modules. Documenting stdlib inter-dependencies may be possible, but I
> wouldn't like to make "X doesn't depend on Y" a guarantee that's
> subject to backward compatibility rules.
>
> Maybe the suggestion is to provide better tools for people wanting to
> *build* such stripped down versions? That might be a possibility, I
> guess. I don't know much about how people build their own copies of
> Python to be able to comment.
>
> So I guess my question is, what is the actual proposal here? People
> seem to have concerns over things that aren't actually being proposed
> - but without knowing what *is* being proposed, it's hard to avoid
> that.
>

Realizing that all of these are just proposals with no solid plan behind
them, they are all predicated on moving to GitHub, and none of these are
directly promoting releasing every module in the stdlib on PyPI as a
stand-alone package with its own versioning, they are:

   1.  Break the stdlib out from CPython and have it be a stand-alone repo
   2. Break the stdlib up into a bunch of independent repos that when
   viewed together make up the stdlib (Steve Dower did some back-of-envelope
   grouping and pegged the # of repos at ~50)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Webmaster] Unsafe listing by Norton's "File Insight"

[Peter via Python-Dev]

(Response at bottom)


On 6/07/2016 2:39 AM, Steve Dower wrote:

On 04Jul2016 2241, Steve Holden wrote:

Hi Peter,

While the humble webmasters can do little about this it's possible the
developers can, so I am forwarding your email to their mailing list.

regards
 Steve

Steve Holden

On Tue, Jul 5, 2016 at 3:30 AM, Peter via Webmaster
mailto:webmas...@python.org>> wrote:

Hi
I'm a heavy user of Python on Windows, am a Basic PSF member and
have contributed to core Python.
The Python 2.7.12 Windows installer download is being marked as
untrusted by Norton Internet Security. I've been on chat with
Symantec, and they've said that I can't do anything about that
rating, but that the site owner can.
I've been pointed to:
https://safeweb.norton.com/help/site_owners
Interestingly, the 3.5.2 download is flagged as safe.
Hoping to get more Python out to users!
Thanks
Peter


Peter, can you provide the exact URL that safeweb is complaining 
about? I tried a few at https://safeweb.norton.com/ and they all 
showed up as clean.


Also please clarify whether this is what you mean. It's not entirely 
clear whether the download is being scanned or the reputation of the 
URL is in question.


Cheers,
Steve

Hi Steve
It's not the URL it is complaining of, rather
On Windows, Norton Internet Security virus checks all downloads. One of 
the names they give to the result of their scanning is 'File Insight'. 
From what I can tell, it uses a few checks:

- virus scanning using known signatures
- observable malicious behaviour
- how well known or used it is across other users of Nortons.
It seems that the last of these is a causing the warning.
Obviously this is not a problem for me, but may be concerning for less 
tech savvy Windows users that get the warning.
There isn't a way within Nortons to 'report for clearance' of the file. 
And my chat with a (entry level) Norton representative got nowhere.
I'll email screen captures in the next email. Let me know if they don't 
come through and I'll paste them somewhere.

Let me know if I can give any more information.
Peter

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Steve Dower]

On 05Jul2016 1404, Brett Cannon wrote:

Realizing that all of these are just proposals with no solid plan behind
them, they are all predicated on moving to GitHub, and none of these are
directly promoting releasing every module in the stdlib on PyPI as a
stand-alone package with its own versioning, they are:

 1.  Break the stdlib out from CPython and have it be a stand-alone repo
 2. Break the stdlib up into a bunch of independent repos that when
viewed together make up the stdlib (Steve Dower did some
back-of-envelope grouping and pegged the # of repos at ~50)


Actually, I was meaning to directly promote releasing each module on 
PyPI as a standalone package :)


"Each module" here is at most the ~50 I counted (though likely many many 
fewer), which sounds intimidating until you realise that there are 
virtually no cross-dependencies between them and most would only depend 
on the core stdlib (which would *not* be a package on PyPI - you get 
most of Lib/*.py with the core install and it's fixed, while much of 
Lib/**/* is separately updateable).


Take email as an example. It's external dependencies (found by grep for 
"import") are abc, base64, calendar, datetime, functools, imghdr, os, 
quopri, sndhdr, socket, time, urllib.parse, uu, warnings. IMHO, only 
urllib has the slightest chance of being non-fixed here (remembering 
that "non-fixed" means upgradeable from PyPI, not that it's missing). 
The circular references (email<->urllib) would probably need to be 
resolved, but I think many would see that as a good cleanliness step anyway.


A quick glance suggests that both email and urllib are only using each 
other's public APIs, which means that any version of the other package 
is sufficient. An official release picks the two latest designated 
stable releases (this is where I'm imagining a requirements.txt-like 
file in the core repository) and bundles them both, and then users can 
update either package on its own. If email makes a change that requires 
a particular change to urllib, it adds a version constraint, which will 
force both users and the core requirements.txt file to update both 
together (this is probably why the circular references would need 
breaking...)


Done with care and even incrementally (e.g. just the provisional modules 
at first), I don't think this is that scary a proposition. It's not 
strictly predicated on moving to github or to many repositories, but if 
we did decide to make a drastic change to the repository layout (which I 
think this requires at a minimum, at least for our own sanity), doing it 
while migrating VCS at least keeps all the disruption together.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Webmaster] Unsafe listing by Norton's "File Insight"

[Steve Dower]

On 05Jul2016 1615, Peter wrote:

It's not the URL it is complaining of, rather
On Windows, Norton Internet Security virus checks all downloads. One of
the names they give to the result of their scanning is 'File Insight'.
From what I can tell, it uses a few checks:
- virus scanning using known signatures
- observable malicious behaviour
- how well known or used it is across other users of Nortons.
It seems that the last of these is a causing the warning.
Obviously this is not a problem for me, but may be concerning for less
tech savvy Windows users that get the warning.
There isn't a way within Nortons to 'report for clearance' of the file.
And my chat with a (entry level) Norton representative got nowhere.
I'll email screen captures in the next email. Let me know if they don't
come through and I'll paste them somewhere.
Let me know if I can give any more information.


I don't think there's anything we can do about this, other than convince 
more users of Norton Internet Security to use Python 2.7 (apparently 
Python 3.5 is more popular with that demographic :) )


The installer is signed with the PSF's certificate, which keeps Windows 
Smartscreen happy and is the only way we can indicate that it's 
trustworthy. If Norton requires different criteria then that is on them 
and not something we can fix.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

[Nick Coghlan]

On 6 July 2016 at 03:44, Barry Warsaw  wrote:
> For example, 3.6 final will come out in December 2016, so it'll be past our
> current 16.10 Ubuntu release.  We've pretty much decided to carry Python 3.5
> through until 17.04, and that'll give us a good year to make 18.04 LTS have a
> solid Python 3.6 ecosystem.

This aligns pretty well with Fedora's plans - the typical Fedora
release dates are May & November, so we will stick with 3.5 for this
year's F25 release, while the Fedora 26 Rawhide branch is expected to
switch to 3.6 shortly after the first 3.6 beta is released in
September. The results in Rawhide should thus help with upstream 3.6
beta testing, with the full release of F26 happening in May 2017 or
so.

> Projecting ahead, it probably means 3.7 in mid-2018, which is after the Ubuntu
> 18.04 LTS release, so we'll only do one major transition before the next LTS.
> From my perspective, that feels about right.

Likewise - 24 months is a bit too slow in getting features out, 12
months expands the community version support matrix too much, while 18
months means that even folks supporting 5* year old LTS Linux releases
will typically only be a couple of releases behind the latest version.

Cheers,
Nick.

* For folks that don't closely follow the way enterprise Linux distros
work, the '5' there isn't arbitrary - it's the lifecycle of Ubuntu LTS
releases, and roughly the length of the "Production 1" phase of RHEL
releases (where new features may still be added in point releases).
Beyond the 5 year mark, I don't think it's particularly reasonable for
people to expect free community support, as even Red Hat stops
backporting anything other than bug fixes and hardware driver updates
at that point. Regardless of your choice of LTS platform, newer
versions will be available by the time your current one is that old,
so "I don't want to upgrade" is a privilege people can reasonably be
expected to pay for.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Webmaster] Unsafe listing by Norton's "File Insight"

[Peter via Python-Dev]

On 6/07/2016 9:51 AM, Steve Dower wrote:

On 05Jul2016 1615, Peter wrote:

It's not the URL it is complaining of, rather
On Windows, Norton Internet Security virus checks all downloads. One of
the names they give to the result of their scanning is 'File Insight'.
From what I can tell, it uses a few checks:
- virus scanning using known signatures
- observable malicious behaviour
- how well known or used it is across other users of Nortons.
It seems that the last of these is a causing the warning.
Obviously this is not a problem for me, but may be concerning for less
tech savvy Windows users that get the warning.
There isn't a way within Nortons to 'report for clearance' of the file.
And my chat with a (entry level) Norton representative got nowhere.
I'll email screen captures in the next email. Let me know if they don't
come through and I'll paste them somewhere.
Let me know if I can give any more information.


I don't think there's anything we can do about this, other than 
convince more users of Norton Internet Security to use Python 2.7 
(apparently Python 3.5 is more popular with that demographic :) )


The installer is signed with the PSF's certificate, which keeps 
Windows Smartscreen happy and is the only way we can indicate that 
it's trustworthy. If Norton requires different criteria then that is 
on them and not something we can fix.


Cheers,
Steve


I suspect you're right.
It's a flawed model that they're using, and they are quite impervious to 
suggestions.

Glad 3.5 is winning :-)
Keep up the good work.
Peter

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

[Nick Coghlan]

On 6 July 2016 at 05:11, Brett Cannon  wrote:
> Sticking w/ 18 months is also fine, but then I would like to discuss
> choosing what months we try to release to get into a date-based release
> cadence so we know that every e.g. December and June are when releases
> typically happen thanks to our 6 month bug-fix release cadence. This has the
> nice benefit of all of us being generally aware of when a bug-fix release is
> coming up instead of having to check the PEP or go through our mail archive
> to find out what month a bug-fix is going to get cut (and also something the
> community to basically count on).

I don't have a strong preference on that front, as even the worst case
outcome of a schedule misalignment for Fedora is what's happening for
Fedora 25 & 26: F25 in November will still have Python 3.5, while
Rawhide will get the 3.6 beta in September or so and then F26 will be
released with 3.6 in the first half of next year.

So really, I think the main criterion here is "Whatever works best for
the folks directly involved in release management"

However, if we did decide we wanted to take minimising "time to
redistribution" for at least Ubuntu & Fedora into account, then the
two main points to consider would be:

- starting the upstream beta phase before the first downstream alpha freeze
- publishing the upstream final release before the last downstream beta freeze

Assuming 6 month distro cadences, and taking the F25 and 16.10 release
cycles as representative, we get:

- Ubuntu alpha 1 releases in late January & June
- Fedora alpha freezes in early February & August
- Ubuntu final beta freezes in late March & September
- Fedora beta freezes in late March & September

Further assuming we stuck with the current model of ~3 months from
beta 1 -> final release, that would suggest a cadence alternating
between:

* December beta, February release
* May beta, August release

If we did that, then 3.6 -> 3.7 would be another "short" cycle (15
months from Dec 2016 to Feb 2018) before settling into a regular
cadence of:

* 2017-12: 3.7.0b1
* 2018-02: 3.7.0
* 2019-05: 3.8.0b1
* 2019-08: 3.8.0
* 2020-12: 3.9.0b1
* 2021-02: 3.9.0
* 2022-05: 3.10.0b1
* 2022-08: 3.10.0
* etc...

The precise timing of maintenance releases isn't as big a deal (since
they don't require anywhere near as much downstream coordination), but
offsetting them by a month from the feature releases (so March &
September in the Fedbuntu driven proposal above) would allow for the
X.(Y+1).1 release to go out at the same time as the final X.Y.Z
release.

I'll reiterate though that we should be able to adjust to *any*
consistent 18 month cycle downstream - the only difference will be the
typical latency between new versions being released on python.org, and
them showing up in Linux distros as the system Python installation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

[Barry Warsaw]

On Jul 06, 2016, at 10:02 AM, Nick Coghlan wrote:

>On 6 July 2016 at 03:44, Barry Warsaw  wrote:
>
>> Projecting ahead, it probably means 3.7 in mid-2018, which is after the
>> Ubuntu 18.04 LTS release, so we'll only do one major transition before the
>> next LTS.  From my perspective, that feels about right.
>
>Likewise - 24 months is a bit too slow in getting features out, 12
>months expands the community version support matrix too much, while 18
>months means that even folks supporting 5* year old LTS Linux releases
>will typically only be a couple of releases behind the latest version.

Cool.  Not that there aren't other distros and OSes involved, but having at
least this much alignment is a good sign.

I should also note that while Debian has a release-when-ready approach, Python
3.6 alpha 2-ish is available in Debian experimental for those who like the
bleeding edge.

Cheers,
-Barry


pgpIEwDK9qkBf.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

[Barry Warsaw]

On Jul 06, 2016, at 10:55 AM, Nick Coghlan wrote:

>However, if we did decide we wanted to take minimising "time to
>redistribution" for at least Ubuntu & Fedora into account, then the
>two main points to consider would be:
>
>- starting the upstream beta phase before the first downstream alpha freeze
>- publishing the upstream final release before the last downstream beta freeze

There have been cases in the past where the schedules didn't align perfectly,
and we really wanted to get ahead of the game, so we released with a late
beta, and then got SRUs (stable release upgrade approval) to move to the final
release *after* the Ubuntu final release.  This isn't great though, especially
for non-LTS releases because they have shorter lifecycles and no point
releases.

Cheers,
-Barry


pgpVIR8scDDik.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence

[Nick Coghlan]

On 6 July 2016 at 03:47, Barry Warsaw  wrote:
> On Jul 05, 2016, at 10:38 AM, Steve Dower wrote:
>
>>My hope is that it would be essentially a "pip freeze"/"pip install -r ..."
>>(or equivalent with whatever tool is used/created for managing the
>>stdlib). Perhaps using VCS URIs rather than version numbers?
>>
>>That is, the test run would dump a list of exactly which stdlib versions it's
>>using, so that when you review the results it is possible to recreate it.
>
> I think you'd have to have vcs checkouts though, because you will often need
> to fix or change something in one of those other library pieces.  The other
> complication of course is that now you'll have two dependent PRs with reviews
> in two different repos.

I'd actually advocate for keeping a unified clone, and make any use of
pip to manage pieces of the standard library purely an install-time
thing (as it is for ensurepip).

The main problem I see with actually making stdlib development
dependent on having a venv already set up for pip to do its thing
without affecting the rest of your system is that it would pose a
major bootstrapping problem.

It does mean we'd be introducing a greater divergence between the way
devs work locally and the way the CI system worked (as in this model
we'd definitely want the buildbots to be exercising both the "test in
checkout" and "test an installed version" case), but working across
multiple repos would be worse.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Nick Coghlan]

On 6 July 2016 at 07:04, Brett Cannon  wrote:
> Realizing that all of these are just proposals with no solid plan behind
> them, they are all predicated on moving to GitHub, and none of these are
> directly promoting releasing every module in the stdlib on PyPI as a
> stand-alone package with its own versioning, they are:
>
> 1. Break the stdlib out from CPython and have it be a stand-alone repo
> 2. Break the stdlib up into a bunch of independent repos that when viewed
> together make up the stdlib (Steve Dower did some back-of-envelope grouping
> and pegged the # of repos at ~50)

3. Keep everything in the main CPython repo, but add a "Bundled"
subdirectory of independently releasable multi-version compatible
subprojects that we move some Lib/* components to.

I think one of our goals here should be that "./configure && make &&
make altinstall" continues to get you a full Python installation for
the relevant version.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

[Nick Coghlan]

On 6 July 2016 at 11:09, Barry Warsaw  wrote:
> On Jul 06, 2016, at 10:55 AM, Nick Coghlan wrote:
>
>>However, if we did decide we wanted to take minimising "time to
>>redistribution" for at least Ubuntu & Fedora into account, then the
>>two main points to consider would be:
>>
>>- starting the upstream beta phase before the first downstream alpha freeze
>>- publishing the upstream final release before the last downstream beta freeze
>
> There have been cases in the past where the schedules didn't align perfectly,
> and we really wanted to get ahead of the game, so we released with a late
> beta, and then got SRUs (stable release upgrade approval) to move to the final
> release *after* the Ubuntu final release.  This isn't great though, especially
> for non-LTS releases because they have shorter lifecycles and no point
> releases.

Aye, Petr and I actually discussed doing something like that in order
to get Python 3.6 into F25, but eventually decided it would be better
to just wait the extra 6 months. We may end up creating a Python 3.6
COPR for F24 & 25 though, similar to the one Matej Stuchlik created
for F23 when Python 3.5 didn't quite make the release deadlines.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Brett Cannon]

On Tue, 5 Jul 2016 at 18:16 Nick Coghlan  wrote:

> On 6 July 2016 at 07:04, Brett Cannon  wrote:
> > Realizing that all of these are just proposals with no solid plan behind
> > them, they are all predicated on moving to GitHub, and none of these are
> > directly promoting releasing every module in the stdlib on PyPI as a
> > stand-alone package with its own versioning, they are:
> >
> > 1. Break the stdlib out from CPython and have it be a stand-alone repo
> > 2. Break the stdlib up into a bunch of independent repos that when viewed
> > together make up the stdlib (Steve Dower did some back-of-envelope
> grouping
> > and pegged the # of repos at ~50)
>
> 3. Keep everything in the main CPython repo, but add a "Bundled"
> subdirectory of independently releasable multi-version compatible
> subprojects that we move some Lib/* components to.
>

That's basically what Steve is proposing.


>
> I think one of our goals here should be that "./configure && make &&
> make altinstall" continues to get you a full Python installation for
> the relevant version.
>

I don't think anyone is suggesting otherwise. You just might have to do
`git clone --recursive` to get a full-fledged CPython checkout w/ stdlib.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

[Steven D'Aprano]

On Tue, Jul 05, 2016 at 08:01:43PM +0200, Petr Viktorin wrote:

> In the tkinter case, compiling for source is easy on a developer's
> computer, but doing that on a headless server brings in devel files for
> the entire graphical environment.
> Are you saying Python on servers should have a way to do turtle
> graphics, otherwise it's not Python?

That's a really good question.

I don't think we have an exact answer to "What counts as Python?". It's 
not like EMCAScript (Javascript) or C where there's a standard that 
defines the language and standard modules. We just have some de facto 
guidelines:

- CPython is definitely Python;
- Jython is surely Python, even if it lacks the byte-code of CPython and 
  some things behave slightly differently;
- MicroPython is probably Python, because nobody expects to be able to 
  run Tkinter GUI apps on an embedded device with 256K or RAM;

but it's hard to make that judgement except on a case-by-case basis.

I think though that even if there's no documented line, most people 
recognise that there are "core" and "non-core" standard modules. dis and 
tkinter are non-core: if µPython leaves out tkinter, nobody will be 
surprised; if Jython leaves out dis, nobody will hold it against them; 
but if they leave out math or htmllib that's another story.

So a headless server can probably leave out tkinter; but a desktop 
shouldn't.


[...]
> > The other extreme is Javascript/Node.js, where the "just use pip" (or 
> > npm in this case) philosophy has been taken to such extremes that one 
> > developer practically brought down the entire Node.js ecosystem by 
> > withdrawing an eleven line module, left-pad, in a fit of pique.
> > 
> > Being open source, the damage was routed around quite quickly, but 
> > still, I think it's a good cautionary example of how a technological 
> > advance can transform a programming culture to the worse.
> 
> I don't understand the analogy. Should the eleven-line module have been
> in Node's stdlib? Outside of stdlib, people are doing this.

The point is that Javascript/Node.js is so lacking in batteries that the 
community culture has gravitated to an extreme version of "just use 
pip". I'm not suggesting that you, or anyone else, has proposed that 
Python do the same, only that there's a balance to be found between the 
extremes of "everything in the Python ecosystem should be part of the 
standard installation" and "next to nothing should be part of the 
standard installation".

The hard part is deciding where that balance should be :-)


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com