[Numpy-discussion] deprecate updateifcopy in nditer operand flags?
I filed issue 9714 https://github.com/numpy/numpy/issues/9714 and wrote
a mail in September trying to get some feedback on what to do with
updateifcopy semantics and user-exposed nditer.
It garnered no response, so I am trying again.
For those who are unfamiliar with the issue see below for a short
summary and issue 7054 for a lengthy discussion.
Note that pull request 9639 which should be merged very soon changes the
magical UPDATEIFCOPY into WRITEBACKIFCOPY, and hopefully will appear in
NumPy 1.14.
As I mention in the issue, there is a magical update done in this
snippet in the next-to-the-last line:
|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T i = np.nditer(a, [],
[['readwrite', 'updateifcopy']], casting='same_kind',
op_dtypes=[np.dtype('f4')]) # Check that UPDATEIFCOPY is activated
i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 i = None #
magic!!! assert a[2, 1, 1] == -12.5|
Not only is this magic very implicit, it relies on refcount semantics
and thus does not work on PyPy.
Possible solutions:
1. nditer is rarely used, just deprecate updateifcopy use on operands
2. make nditer into a context manager, so the code would become explicit
|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T with np.nditer(a, [],
[['readwrite', 'updateifcopy']], casting='same_kind',
op_dtypes=[np.dtype('f4')]) as i: # Check that WRITEBACKIFCOPY is
activated i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5
assert a[2, 1, 1] == -12.5 # a is modified in i.__exit__|
3. something else?
Any opinions? Does anyone use nditer in production code?
Matti
-
what are updateifcopy semantics? When a temporary copy or work buffer is
required, NumPy can (ab)use the base attribute of an ndarray by
- creating a copy of the data from the base array
- mark the base array read-only
Then when the temporary buffer is "no longer needed"
- the data is copied back
- the original base array is marked read-write
The trigger for the "no longer needed" decision before pull request 9639
is in the dealloc function.
That is not generally a place to do useful work, especially on PyPy
which can call dealloc much later.
Pull request 9639 adds an explicit PyArray_ResolveWritebackIfCopy api
function, and recommends calling it explicitly before dealloc.
The only place this change is visible to the python-level user is in
nditer.
C-API users will need to adapt their code to use the new API function,
with a deprecation cycle that is backwardly compatible on CPython.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Tue, Nov 7, 2017 at 11:40 PM, Nathaniel Smith wrote: > > > > Right now, the decision in front of us is what to tell people who ask about > numpy's py2 support plans, so that they can make their own plans. Given what > we know right now, I don't think we should promise to keep support past > 2018. If we get there and the situation's changed, and there's both desire > and means to extend support we can revisit that. But's better to > under-promise and possibly over-deliver, instead of promising to support py2 > until after it becomes a millstone around our necks and then realizing we > haven't warned anyone and are stuck supporting it another year beyond > that... > > -n NumPy (and to a lesser extent SciPy) is in a tough position being at the bottom of many scientific Python programming stacks. Whenever you drop Python 2 support is going to upset someone. It is too ambitious to pledge to drop support for Python 2.7 no later than 2020, coinciding with the Python development team’s timeline for dropping support for Python 2.7? If that looks doable, NumPy could sign up to http://www.python3statement.org/ Regards, Peter ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
I was about to send the same thing. I think this matter became a vim/emacs issue and Py2 supporters won't take any arguments anymore. But if Instagram can do it, it means that legacy code argument is a matter of will but not a technicality. https://thenewstack.io/instagram-makes-smooth-move-python-3/ Also people are really going out of their ways such as Tauthon https://github.com/naftaliharris/tauthon to stay with Python2. To be honest, I'm convinced that this is a sentimental debate after seeing this fork. On Wed, Nov 8, 2017 at 5:50 PM, Peter Cock wrote: > On Tue, Nov 7, 2017 at 11:40 PM, Nathaniel Smith wrote: > > > > > > > > Right now, the decision in front of us is what to tell people who ask > about > > numpy's py2 support plans, so that they can make their own plans. Given > what > > we know right now, I don't think we should promise to keep support past > > 2018. If we get there and the situation's changed, and there's both > desire > > and means to extend support we can revisit that. But's better to > > under-promise and possibly over-deliver, instead of promising to support > py2 > > until after it becomes a millstone around our necks and then realizing we > > haven't warned anyone and are stuck supporting it another year beyond > > that... > > > > -n > > NumPy (and to a lesser extent SciPy) is in a tough position being at the > bottom of many scientific Python programming stacks. Whenever you > drop Python 2 support is going to upset someone. > > It is too ambitious to pledge to drop support for Python 2.7 no later than > 2020, coinciding with the Python development team’s timeline for dropping > support for Python 2.7? > > If that looks doable, NumPy could sign up to http://www.python3statement. > org/ > > Regards, > > Peter > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] deprecate updateifcopy in nditer operand, flags?
Date: Wed, 8 Nov 2017 18:41:03 +0200
From: Matti Picus
To: numpy-discussion@python.org
Subject: [Numpy-discussion] deprecate updateifcopy in nditer operand
flags?
Message-ID:
Content-Type: text/plain; charset=utf-8; format=flowed
I filed issue 9714 https://github.com/numpy/numpy/issues/9714 and wrote
a mail in September trying to get some feedback on what to do with
updateifcopy semantics and user-exposed nditer.
It garnered no response, so I am trying again.
For those who are unfamiliar with the issue see below for a short
summary and issue 7054 for a lengthy discussion.
Note that pull request 9639 which should be merged very soon changes the
magical UPDATEIFCOPY into WRITEBACKIFCOPY, and hopefully will appear in
NumPy 1.14.
As I mention in the issue, there is a magical update done in this
snippet in the next-to-the-last line:
|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T i = np.nditer(a, [],
[['readwrite', 'updateifcopy']], casting='same_kind',
op_dtypes=[np.dtype('f4')]) # Check that UPDATEIFCOPY is activated
i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 i = None #
magic!!! assert a[2, 1, 1] == -12.5|
Formatting
a = np.arange(24, dtype='f8').reshape(2, 3, 4).T
i = np.nditer(a, [], [['readwrite', 'updateifcopy']],
casting='same_kind',
op_dtypes=[np.dtype('f4')])
# Check that WRITEBACKIFCOPY is activated
i.operands[0][2, 1, 1] = -12.5
assert a[2, 1, 1] != -12.5
i=None # magic
assert a[2, 1, 1] == -12.5
Not only is this magic very implicit, it relies on refcount semantics
and thus does not work on PyPy.
Possible solutions:
1. nditer is rarely used, just deprecate updateifcopy use on operands
2. make nditer into a context manager, so the code would become explicit
|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T with np.nditer(a, [],
[['readwrite', 'updateifcopy']], casting='same_kind',
op_dtypes=[np.dtype('f4')]) as i: # Check that WRITEBACKIFCOPY is
activated i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5
assert a[2, 1, 1] == -12.5 # a is modified in i.__exit__|
Formatting
a = np.arange(24, dtype='f8').reshape(2, 3, 4).T
with np.nditer(a, [], [['readwrite', 'updateifcopy']],
casting='same_kind',
op_dtypes=[np.dtype('f4')]) as i:
# Check that WRITEBACKIFCOPY is activated
i.operands[0][2, 1, 1] = -12.5
assert a[2, 1, 1] != -12.5
assert a[2, 1, 1] == -12.5 # a is modified in
i.__exit__
3. something else?
Any opinions? Does anyone use nditer in production code?
Matti
-
what are updateifcopy semantics? When a temporary copy or work buffer is
required, NumPy can (ab)use the base attribute of an ndarray by
?? - creating a copy of the data from the base array
?? - mark the base array read-only
Then when the temporary buffer is "no longer needed"
?? - the data is copied back
?? - the original base array is marked read-write
The trigger for the "no longer needed" decision before pull request 9639
is in the dealloc function.
That is not generally a place to do useful work, especially on PyPy
which can call dealloc much later.
Pull request 9639 adds an explicit PyArray_ResolveWritebackIfCopy api
function, and recommends calling it explicitly before dealloc.
The only place this change is visible to the python-level user is in
nditer.
C-API users will need to adapt their code to use the new API function,
with a deprecation cycle that is backwardly compatible on CPython.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Wed, 2017-11-08 at 18:15 +0100, Ilhan Polat wrote: > I was about to send the same thing. I think this matter became a > vim/emacs issue and Py2 supporters won't take any arguments anymore. > But if Instagram can do it, it means that legacy code argument is a > matter of will but not a technicality. https://thenewstack.io/instagr > am-makes-smooth-move-python-3/ > > Also people are really going out of their ways such as Tauthon https: > //github.com/naftaliharris/tauthon to stay with Python2. To be > honest, I'm convinced that this is a sentimental debate after seeing > this fork. > > In my opinion it is fine for us to drop support for python 2 in master relatively soon (as proposed here). But I guess we will need to a "LTS" release which means some extra maintenance burden until 2020. I could hope those who really need it jumping in to carry some of that (and by 2020 my guess is if anyone still wants to support it longer, we won't stop you, but I doubt the current core devs, at least not me, would be very interested in it). So in my opinion, the current NumPy is excellent and very stable, anyone who needs fancy new stuff likely also wants other fancy new stuff so will soon have to use python 3 anyway Which means, if we think the extra burden of a "LTS" is lower then the current hassle, lets do it :). Also downstream seems only half a reason to me, since downstream normally supports much outdated versions anyway? - Sebastian > > > > > > On Wed, Nov 8, 2017 at 5:50 PM, Peter Cock > wrote: > > On Tue, Nov 7, 2017 at 11:40 PM, Nathaniel Smith > > wrote: > > > > > > > > > > > > Right now, the decision in front of us is what to tell people who > > ask about > > > numpy's py2 support plans, so that they can make their own plans. > > Given what > > > we know right now, I don't think we should promise to keep > > support past > > > 2018. If we get there and the situation's changed, and there's > > both desire > > > and means to extend support we can revisit that. But's better to > > > under-promise and possibly over-deliver, instead of promising to > > support py2 > > > until after it becomes a millstone around our necks and then > > realizing we > > > haven't warned anyone and are stuck supporting it another year > > beyond > > > that... > > > > > > -n > > > > NumPy (and to a lesser extent SciPy) is in a tough position being > > at the > > bottom of many scientific Python programming stacks. Whenever you > > drop Python 2 support is going to upset someone. > > > > It is too ambitious to pledge to drop support for Python 2.7 no > > later than > > 2020, coinciding with the Python development team’s timeline for > > dropping > > support for Python 2.7? > > > > If that looks doable, NumPy could sign up to http://www.python3stat > > ement.org/ > > > > Regards, > > > > Peter > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On 06.11.2017 11:10, Ralf Gommers wrote: > > > On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris > mailto:charlesr.har...@gmail.com>> wrote: > > Hi All, > > Thought I'd toss this out there. I'm tending towards better sooner > than later in dropping Python 2.7 support as we are starting to run > up against places where we would like to use Python 3 features. That > is particularly true on Windows where the 2.7 compiler is really old > and lacks C99 compatibility. > > > This is probably the most pressing reason to drop 2.7 support. We seem > to be expending a lot of effort lately on this stuff. I was previously > advocating being more conservative than the timeline you now propose, > but this is the pain point that I think gets me over the line. Would dropping python2 support for windows earlier than the other platforms a reasonable approach? I am not a big fan of to dropping python2 support before 2020, but I have no issue with dropping python2 support on windows earlier as it is our largest pain point. ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] deprecate updateifcopy in nditer operand, flags?
At a higher level:
The issue here is that we need to break the nditer API. This might
affect you if you np.nditer (in Python) or the NpyIter_* APIs (in C).
The exact cases affected are somewhat hard to describe because
nditer's flag processing is complicated [1], but basically it's cases
where you are writing to one of the arrays being iterated over and
then something else non-trivial happens.
The problem is that the API currently uses NumPy's odd UPDATEIFCOPY
feature. What it does is give you an "output" array which is not your
actual output array, but instead some other temporary array which you
can modify freely, and whose contents are later written back to your
actual output array.
When does this copy happen? Since this is an iterator, then most of
the time we can do the writeback for iteration N when we start
iteration N+1. However, this doesn't work for the final iteration. On
the final iteration, currently the writeback happens when the
temporary is garbage collected. *Usually* this happens pretty
promptly, but this is dependent on some internal details of how
CPython's garbage collector works that are explicitly not part of the
Python language spec, and on PyPy you silently and
non-deterministically get incorrect results. Plus it's error-prone
even on CPython -- if you accidentally have a dangling reference to
one array, then suddenly another array will have the wrong contents.
So we have two options:
- We could stop supporting this mode entirely. Unfortunately, it's
hard to know if anyone is using this, since the conditions to trigger
it are so complicated, and not necessarily very exotic (e.g. it can
happen if you have a function that uses nditer to read one array and
write to another, and then someone calls your function with two arrays
whose memory overlaps).
- We could adjust the API so that there's some explicit operation to
trigger the final writeback. At the Python level this would probably
mean that we start supporting the use of nditer as a context manager,
and eventually start raising an error if you're in one of the "unsafe"
case and not using the context manager form. At the C level we
probably need some explicit "I'm done with this iterator now" call.
One question is which cases exactly should produce warnings/eventually
errors. At the Python level, I guess the simplest rule would be that
if you have any write/readwrite arrays in your iterator, then you have
to use a 'with' block. At the C level, it's a little trickier, because
it's hard to tell up-front whether someone has updated their code to
call a final cleanup function, and it's hard to emit a warning/error
on something that *doesn't* happen. (You could print a warning when
the nditer object is GCed if the cleanup function wasn't called, but
you can't raise an error there.) I guess the only reasonable option is
to deprecate NPY_ITER_READWRITE and NP_ITER_WRITEONLY, and make people
switch to passing new flags that have the same semantics but also
promise that the user has updated their code to call the new cleanup
function.
Does that work? Any objections?
-n
[1] The affected cases are the ones that reach this line:
https://github.com/numpy/numpy/blob/c276f326b29bcb7c851169d34f4767da0b4347af/numpy/core/src/multiarray/nditer_constr.c#L2926
So it's something like
- all of these things are true:
- you have a writable array (nditer flags "write" or "readwrite")
- one of these things is true:
- you passed the "forcecopy" flag
- all of these things are true:
- you requested casting
- you requested updateifcopy
- there's a memory overlap between this array and another of the
arrays being iterated over
On Wed, Nov 8, 2017 at 11:31 AM, Matti Picus wrote:
>
> Date: Wed, 8 Nov 2017 18:41:03 +0200
> From: Matti Picus
> To: numpy-discussion@python.org
> Subject: [Numpy-discussion] deprecate updateifcopy in nditer operand
> flags?
> Message-ID:
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> I filed issue 9714 https://github.com/numpy/numpy/issues/9714 and wrote
> a mail in September trying to get some feedback on what to do with
> updateifcopy semantics and user-exposed nditer.
> It garnered no response, so I am trying again.
> For those who are unfamiliar with the issue see below for a short
> summary and issue 7054 for a lengthy discussion.
> Note that pull request 9639 which should be merged very soon changes the
> magical UPDATEIFCOPY into WRITEBACKIFCOPY, and hopefully will appear in
> NumPy 1.14.
>
> As I mention in the issue, there is a magical update done in this
> snippet in the next-to-the-last line:
>
> |a = np.arange(24, dtype='f8').reshape(2, 3, 4).T i = np.nditer(a, [],
> [['readwrite', 'updateifcopy']], casting='same_kind',
> op_dtypes=[np.dtype('f4')]) # Check that UPDATEIFCOPY is activated
> i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 i = None #
> magic!!! assert a[2, 1, 1] == -12.5|
>
> Formatting
>
> a = np.arange(24, dtype='f8').reshape(2, 3, 4).T
>
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Wed, Nov 8, 2017 at 11:08 AM, Julian Taylor < jtaylor.deb...@googlemail.com> wrote: > > Would dropping python2 support for windows earlier than the other > platforms a reasonable approach? > no. I'm not Windows fan myself, but it is a HUGE fraction of the userbase. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] deprecate updateifcopy in nditer operand, flags?
On 11/08/2017 03:12 PM, Nathaniel Smith wrote: > - We could adjust the API so that there's some explicit operation to > trigger the final writeback. At the Python level this would probably > mean that we start supporting the use of nditer as a context manager, > and eventually start raising an error if you're in one of the "unsafe" > case and not using the context manager form. At the C level we > probably need some explicit "I'm done with this iterator now" call. > > One question is which cases exactly should produce warnings/eventually > errors. At the Python level, I guess the simplest rule would be that > if you have any write/readwrite arrays in your iterator, then you have > to use a 'with' block. At the C level, it's a little trickier, because > it's hard to tell up-front whether someone has updated their code to > call a final cleanup function, and it's hard to emit a warning/error > on something that *doesn't* happen. (You could print a warning when > the nditer object is GCed if the cleanup function wasn't called, but > you can't raise an error there.) I guess the only reasonable option is > to deprecate NPY_ITER_READWRITE and NP_ITER_WRITEONLY, and make people > switch to passing new flags that have the same semantics but also > promise that the user has updated their code to call the new cleanup > function. Seems reasonable. When people use the Nditer C-api, they (almost?) always call NpyIter_Dealloc when they're done. Maybe that's a place to put a warning for C-api users. I think you can emit a warning there since that function calls the GC, not the other way around. It looks like you've already discussed the possibilities of putting things in NpyIter_Dealloc though, and it could be tricky, but if we only need a warning maybe there's a way. https://github.com/numpy/numpy/pull/9269/files/6dc0c65e4b2ea67688d6b617da3a175cd603fc18#r127707149 Allan ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
> On Nov 8, 2017, at 10:50, Peter Cock wrote: > > NumPy (and to a lesser extent SciPy) is in a tough position being at the > bottom of many scientific Python programming stacks. Whenever you > drop Python 2 support is going to upset someone. Existing versions of NumPy will still exist and continue to work with Python 2.7. If users want to say with Python 2.7, that's fine, they will just have to rely on those older/LTS versions. I personally would be happy for projects at the bottom of stacks to take an activist stance and make decisions to actively encourage movement to Python 3. > It is too ambitious to pledge to drop support for Python 2.7 no later than > 2020, coinciding with the Python development team’s timeline for dropping > support for Python 2.7? Developing NumPy is hard, as it is. Everything that can be done to simplify things for the current maintainers and help attract new contributors should be done. It is not reasonable to ask a few (largely volunteer) people to shoulder the burden and difficulties of supporting Python 2.7 for several additional *years* of their life. I agree entirely with Nick Coghlan's comments from another discussion, and think they apply equally well in this instance: """ While it's entirely admirable that many upstream developers are generous enough to help their end users work around this inertia, in the long run doing so is detrimental for everyone concerned, as long term sustaining engineering for old releases is genuinely demotivating for upstream developers (it's a good job, but a lousy way to spend your free time) and for end users, working around institutional inertia this way reduces the pressure to actually get the situation addressed properly. """ Thanks, Bryan ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
Anaconda's compilers are for Linux (gcc 7.2) and Mac (llvm/clang 4.0.1) right now. We would like to have clang target all platforms, but that's a lot of development effort. We are also exploring ways of keeping package ecosystems in line, so that building and managing a self-consistent set of python 2.7 packages with a new Visual Studio version or msys2 might be easier. No timeline to report, on that, though. Breaking with the python.org ABI is pretty painful. On Wed, Nov 8, 2017 at 4:17 PM, Bryan Van de ven wrote: > > > On Nov 8, 2017, at 10:50, Peter Cock wrote: > > > > NumPy (and to a lesser extent SciPy) is in a tough position being at the > > bottom of many scientific Python programming stacks. Whenever you > > drop Python 2 support is going to upset someone. > > Existing versions of NumPy will still exist and continue to work with > Python 2.7. If users want to say with Python 2.7, that's fine, they will > just have to rely on those older/LTS versions. I personally would be happy > for projects at the bottom of stacks to take an activist stance and make > decisions to actively encourage movement to Python 3. > > > It is too ambitious to pledge to drop support for Python 2.7 no later > than > > 2020, coinciding with the Python development team’s timeline for dropping > > support for Python 2.7? > > Developing NumPy is hard, as it is. Everything that can be done to > simplify things for the current maintainers and help attract new > contributors should be done. It is not reasonable to ask a few (largely > volunteer) people to shoulder the burden and difficulties of supporting > Python 2.7 for several additional *years* of their life. > > I agree entirely with Nick Coghlan's comments from another discussion, and > think they apply equally well in this instance: > > """ > While it's entirely admirable that many upstream developers are generous > enough to help their end users work around this inertia, in the long run > doing so is detrimental for everyone concerned, as long term sustaining > engineering for old releases is genuinely demotivating for upstream > developers (it's a good job, but a lousy way to spend your free time) and > for end users, working around institutional inertia this way reduces the > pressure to actually get the situation addressed properly. > """ > > Thanks, > > Bryan > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
Hi, On Wed, Nov 8, 2017 at 7:08 PM, Julian Taylor wrote: > On 06.11.2017 11:10, Ralf Gommers wrote: >> >> >> On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris >> mailto:charlesr.har...@gmail.com>> wrote: >> >> Hi All, >> >> Thought I'd toss this out there. I'm tending towards better sooner >> than later in dropping Python 2.7 support as we are starting to run >> up against places where we would like to use Python 3 features. That >> is particularly true on Windows where the 2.7 compiler is really old >> and lacks C99 compatibility. >> >> >> This is probably the most pressing reason to drop 2.7 support. We seem >> to be expending a lot of effort lately on this stuff. I was previously >> advocating being more conservative than the timeline you now propose, >> but this is the pain point that I think gets me over the line. > > > Would dropping python2 support for windows earlier than the other > platforms a reasonable approach? > I am not a big fan of to dropping python2 support before 2020, but I > have no issue with dropping python2 support on windows earlier as it is > our largest pain point. I wonder about this too. I can imagine there are a reasonable number of people using older Linux distributions on which they cannot upgrade to a recent Python 3, but is that likely to be true for Windows? We'd have to make sure we could persuade pypi to give the older version for Windows, by default - I don't know if that is possible. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Nov 8, 2017 16:51, "Matthew Brett" wrote: Hi, On Wed, Nov 8, 2017 at 7:08 PM, Julian Taylor wrote: > On 06.11.2017 11:10, Ralf Gommers wrote: >> >> >> On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris >> mailto:charlesr.har...@gmail.com>> wrote: >> >> Hi All, >> >> Thought I'd toss this out there. I'm tending towards better sooner >> than later in dropping Python 2.7 support as we are starting to run >> up against places where we would like to use Python 3 features. That >> is particularly true on Windows where the 2.7 compiler is really old >> and lacks C99 compatibility. >> >> >> This is probably the most pressing reason to drop 2.7 support. We seem >> to be expending a lot of effort lately on this stuff. I was previously >> advocating being more conservative than the timeline you now propose, >> but this is the pain point that I think gets me over the line. > > > Would dropping python2 support for windows earlier than the other > platforms a reasonable approach? > I am not a big fan of to dropping python2 support before 2020, but I > have no issue with dropping python2 support on windows earlier as it is > our largest pain point. I wonder about this too. I can imagine there are a reasonable number of people using older Linux distributions on which they cannot upgrade to a recent Python 3, My impression is that this is increasingly rare, actually. I believe RHEL is still shipping 2.6 by default, which we've already dropped support for, and if you want RH python then they provide supported 2.7 and 3.latest through exactly the same channels. Ubuntu 14.04 is end-of-life in April 2019, so pretty irrelevant if we're talking about 2019 for dropping support, and 16.04 ships with 3.5. Plus with docker, conda, PPAs, etc., getting a recent python is easier than its ever been. > but is that likely to be true for Windows? We'd have to make sure we could persuade pypi to give the older version for Windows, by default - I don't know if that is possible. Currently it's not – if pip doesn't see a Windows wheel, it'll try downloading and building an sdist. There's a mechanism for sdists to declare what version of python they support but (thanks to the jupyter folks for implementing this), but that's all. The effect is that if we release a version that drops support for py2 entirely, then 'pip install' on py2 will continue to work and give the last supported version, but if we release a version that drops py2 on Windows but keeps it on other platforms then 'pip install' on py2 on Windows will just stop working entirely. This is possible to fix – it's just software – but I'm not volunteering... -n ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Wed, Nov 8, 2017 at 10:17 PM, Bryan Van de ven wrote: > >> On Nov 8, 2017, at 10:50, Peter Cock wrote: >> >> NumPy (and to a lesser extent SciPy) is in a tough position being at the >> bottom of many scientific Python programming stacks. Whenever you >> drop Python 2 support is going to upset someone. > > Existing versions of NumPy will still exist and continue to work with Python > 2.7. If users want to say with Python 2.7, that's fine, they will just have > to rely on those older/LTS versions. I personally would be happy for projects > at the bottom of stacks to take an activist stance and make decisions to > actively encourage movement to Python 3. > >> It is too ambitious to pledge to drop support for Python 2.7 no later than >> 2020, coinciding with the Python development team’s timeline for dropping >> support for Python 2.7? > > Developing NumPy is hard, as it is. Everything that can be done to simplify > things for the current maintainers and help attract new contributors should > be done. It is not reasonable to ask a few (largely volunteer) people to > shoulder the burden and difficulties of supporting Python 2.7 for several > additional *years* of their life. > > I agree entirely with Nick Coghlan's comments from another discussion, and > think they apply equally well in this instance: > > """ > While it's entirely admirable that many upstream developers are generous > enough to help their end users work around this inertia, in the long run > doing so is detrimental for everyone concerned, as long term sustaining > engineering for old releases is genuinely demotivating for upstream > developers (it's a good job, but a lousy way to spend your free time) and for > end users, working around institutional inertia this way reduces the pressure > to actually get the situation addressed properly. > """ > > Thanks, > > Bryan I agree too - I was trying to phrase that email neutrally as I am not a direct NumPy contributor, but to be more explicit, as someone invested in this ecosystem: I'd fully support NumPy pledging to drop Python 2.7 support no later than 2020. I see signing up to http://www.python3statement.org/ as being about helping publicise this choice. (This is not to say dropping Python 2.7 support in NumPy couldn't happen much sooner than 2020 - the C99 compiler issues sounds like a strong pressure to do so.) Peter ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Thu, Nov 9, 2017 at 12:15 PM, Nathaniel Smith wrote: > On Nov 8, 2017 16:51, "Matthew Brett" wrote: > > Hi, > > On Wed, Nov 8, 2017 at 7:08 PM, Julian Taylor > wrote: > > On 06.11.2017 11:10, Ralf Gommers wrote: > >> > >> > >> On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris > >> mailto:charlesr.har...@gmail.com>> wrote: > >> > >> Hi All, > >> > >> Thought I'd toss this out there. I'm tending towards better sooner > >> than later in dropping Python 2.7 support as we are starting to run > >> up against places where we would like to use Python 3 features. That > >> is particularly true on Windows where the 2.7 compiler is really old > >> and lacks C99 compatibility. > >> > >> > >> This is probably the most pressing reason to drop 2.7 support. We seem > >> to be expending a lot of effort lately on this stuff. I was previously > >> advocating being more conservative than the timeline you now propose, > >> but this is the pain point that I think gets me over the line. > > > > > > Would dropping python2 support for windows earlier than the other > > platforms a reasonable approach? > > I am not a big fan of to dropping python2 support before 2020, but I > > have no issue with dropping python2 support on windows earlier as it is > > our largest pain point. > > I wonder about this too. I can imagine there are a reasonable number > of people using older Linux distributions on which they cannot upgrade > to a recent Python 3, > > > My impression is that this is increasingly rare, actually. I believe RHEL > is still shipping 2.6 by default, which we've already dropped support for, > and if you want RH python then they provide supported 2.7 and 3.latest > through exactly the same channels. Ubuntu 14.04 is end-of-life in April > 2019, so pretty irrelevant if we're talking about 2019 for dropping > support, and 16.04 ships with 3.5. Plus with docker, conda, PPAs, etc., > getting a recent python is easier than its ever been. > > > but > > is that likely to be true for Windows? > > We'd have to make sure we could persuade pypi to give the older > version for Windows, by default - I don't know if that is possible. > > > Currently it's not – if pip doesn't see a Windows wheel, it'll try > downloading and building an sdist. There's a mechanism for sdists to > declare what version of python they support but (thanks to the jupyter > folks for implementing this), but that's all. The effect is that if we > release a version that drops support for py2 entirely, then 'pip install' > on py2 will continue to work and give the last supported version, but if we > release a version that drops py2 on Windows but keeps it on other platforms > then 'pip install' on py2 on Windows will just stop working entirely. > > This is possible to fix – it's just software – but I'm not volunteering... > Given the release cycle of pip (slow) and the bandwidth required to implement this, I think that this is likely a showstopper for Windows-only-3.x-only. Another consideration is that choices made by numpy tend to propagate to the rest of the ecosystem, and support for Python versions that's OS-independent is nicer than Windows special-casing. And yet another is that when we do finally drop 2.7, I think we'd want to get the full benefits of doing so. That's new 3.x features (@ in particular), cleaning up lots of support code, etc. For those reasons I think we should balance the pain and benefits of 2.7 support and just pick a date to drop it completely, not just on Windows. Regarding http://www.python3statement.org/: I'd say that as long as there are people who want to spend their energy on the LTS release (contributors *and* enough maintainer power to review/merge/release), we should not actively prevent them from doing that. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
