[Python-Dev] Re: Advantages of pattern matching - a simple comparative analysis

2021-01-19 Thread Antoine Pitrou 
On Mon, 18 Jan 2021 12:41:45 -0800
Chris Barker via Python-Dev  wrote:
> 
> And in "real world" code, I've done just this -- building a system for
> saving / restoring dataclasses to/from JSON. In that case, each of the
> dataclasses knows how to save itself and build itself from JSON-compatible
> python objects (numbers, dicts, strings, lists) -- so again, no need for
> pattern matching there either. And what I really like about the approach of
> putting all the logic in the "nodes" is that I can make new types of nodes
> without having to touch the code at the "top" that visits those nodes.

Note that another approach is the little-used
`functools.singledispatch`.

Regards

Antoine.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/PD5PIBOLSHL3I7TC7XDCEDKRPRHAJPEY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

2021-01-19 Thread Antoine Pitrou 
On Mon, 18 Jan 2021 15:54:32 -0800
Larry Hastings  wrote:
> On 1/18/21 3:42 PM, Inada Naoki wrote:
> > Many type hinting use cases don't need type objects in runtime.
> > So I think PEP 563 is better for type hinting user experience.  
> 
> You mean, in situations where the user doesn't want to import the types, 
> because of heavyweight imports or circular imports?  I didn't think 
> those were very common.

Probably not very common, but annoying anyway.  For example, a library
(say PyArrow) may expose a function for importing Pandas data without
mandating a Pandas dependency.

Note: I don't use type hinting, I'm just responding to this particular
aspect (optional / heavy dependencies).

Regards

Antoine.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/MV4R3BQCDJZNL6SN2CAAN43EBW5Q6UMF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Please explain how to migrate when a function is removed, thanks ;-)

2021-01-19 Thread Victor Stinner 
Hi,

We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
are facing many Python 3.10 incompatible changes. Most changes were
planned with a deprecation period, but, as usual, projects are not
tested with DeprecationWarning treated as errors, and so tons of
packages now fail to build.

I'm not here to talk about DeprecationWarning, but how we communicate
on incompatible changes made on purpose. For example, What's New in
Python 3.8 announces: "In asyncio, the explicit passing of a loop
argument has been deprecated and will be removed in version 3.10". As
expected, the parameter was removed in Python 3.10 (bpo-42392), but I
cannot see anything in What's New in Python 3.10. The problem is that
I don't know how to fix projects broken by this change. I'm not
complaining about this specific change, but more generally.

I strongly suggest to well document incompatible changes. The bare
minimum is to mention them in "Porting to Python 3.10" section:
https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10

A link to the bpo sometimes helps to understand how to port code. But
I would really appreciate if authors of incompatible changes would
explain how to add Python 3.10 support to existing projects, without
losing support for older Python versions. Not just "this function is
now removed, good luck!" :-)

I didn't touch asyncio for at least 1 year, so I don't know what
happens if I remove a loop argument. Does an application remain
compatible with Python 3.6 without passing loop?

I know that I made multiple incompatible changes myself and I'm sure
that the documentation should probably be enhanced, but I also expect
others to help on that! ;-)

Please think about people who have to port 4000 Python projects to Python 3.10!

---

The best would be to ship a tool which adds Python 3.10 support to
existing projects without losing support with Python 3.6. Maybe
something like https://github.com/asottile/pyupgrade could be used for
that? pyupgrade looks more specific to the Python syntax, than the
usage of the stdlib.

I wrote such tool to add Python 3.10 support to C extensions without
losing support with Python 2.7. It relies on a header file
(pythoncapi_compat.h) which provides new C API functions on old Python
versions.
https://github.com/pythoncapi/pythoncapi_compat

For example, it replaces "obj->ob_refcnt = refcnt;" with
"Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
3.8 and older (function added to Python 3.9)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/O3T7SK3BGMFWMLCQXDODZJSBL42AUWTR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Mark Shannon 

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently 
from runaway recursion. This would allow programs to set the maximum 
recursion depth to fit their needs and provide additional safety guarantees.


The following program will run safely to completion:

sys.setrecursionlimit(1_000_000)

def f(n):
if n:
f(n-1)

f(500_000)

The following program will raise a StackOverflow, without causing a VM 
crash:


sys.setrecursionlimit(1_000_000)

class X:
def __add__(self, other):
return self + other

X() + 1

---

The full PEP can be found here:
https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Terry Reedy 

On 1/19/2021 8:31 AM, Mark Shannon wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently 
from runaway recursion. This would allow programs to set the maximum 
recursion depth to fit their needs and provide additional safety 
guarantees.


The following program will run safely to completion:

     sys.setrecursionlimit(1_000_000)

     def f(n):
     if n:
     f(n-1)

     f(500_000)


Are you sure?  On Windows, after adding the import
and a line at the top of f
if not n % 1000: print(n)
I get with Command Prompt

C:\Users\Terry>py -m a.tem4
50
499000
498000

C:\Users\Terry>

with a pause of after 1 to multiple seconds.  Clearly did not run to 
completion, but no exception or Windows crash box to indicate such 
without the print.


In IDLE, I get nearly the same:
= RESTART: F:\Python\a\tem4.py
50
499000
498000

 RESTART: Shell
>>>
The Shell restart indicates that the user code subprocess crashed and 
was restarted.  I checked that sys.getrecursionlimit() really returns 
1_000_000.


To show completion, do something like add global m and m+=1 in f and m=0 
and print(m) after the f call.



--
Terry Jan Reedy

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/KNWNOKK3QJVS3DCHNY3FOFSVMVU3EVMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Terry Reedy 

On 1/19/2021 8:31 AM, Mark Shannon wrote:


It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently 
from runaway recursion.


My impression is that this is already the case in the sense given below.

https://bugs.python.org/issue42887 is about segfaults with this code 
with no Python-level recursion.


mystr  = "hello123"
for x in range(100):
mystr = mystr.__sizeof__()

Christian Heimes reproduced and said "The stack trace is several hundred 
thousand (!) levels deep."
Ronald Oussoren said "The dealloc of "mystr" will cause recursive calls 
to tp_dealloc along the entire chain and that can exhaust the C stack."


Since hundreds of thousands is a lot bigger than 1000, I assume that the 
C recursion is not tracked.  Whick is to say, recursive C calls are not 
counted, unlike recursive Python calls.  There have been several issues 
about highly nested code and recursive structures causes segfaults, 
presumably due to C level recursion.  These are usually closed as 'Won't 
fix' and the suggestion 'don't do that'.


I strongly agree that a fix would be great.

--
Terry Jan Reedy
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/4NA42RO7E5XV75WHWBEFVSJMFHJ7FCUJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Mark Shannon 



On 19/01/2021 2:38 pm, Terry Reedy wrote:

On 1/19/2021 8:31 AM, Mark Shannon wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently 
from runaway recursion. This would allow programs to set the maximum 
recursion depth to fit their needs and provide additional safety 
guarantees.


The following program will run safely to completion:

 sys.setrecursionlimit(1_000_000)

 def f(n):
 if n:
 f(n-1)

 f(500_000)


Are you sure?  On Windows, after adding the import
and a line at the top of f
     if not n % 1000: print(n)
I get with Command Prompt

C:\Users\Terry>py -m a.tem4
50
499000
498000

C:\Users\Terry>

with a pause of after 1 to multiple seconds.  Clearly did not run to 
completion, but no exception or Windows crash box to indicate such 
without the print.


In IDLE, I get nearly the same:
= RESTART: F:\Python\a\tem4.py
50
499000
498000

 RESTART: Shell
 >>>
The Shell restart indicates that the user code subprocess crashed and 
was restarted.  I checked that sys.getrecursionlimit() really returns 
1_000_000.


To show completion, do something like add global m and m+=1 in f and m=0 
and print(m) after the f call.





I'm not sure whether you are saying that this doesn't work now, that it 
can't work, or that it shouldn't work.


If that it doesn't work now, then I agree. That's why I've written the 
PEP; it should work.


If either of the other two, why?

Cheers,
Mark.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/3F5S7G57GXRZ2C4E7OI5LJSTVGC6NZOI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Antoine Pitrou 
On Tue, 19 Jan 2021 13:31:45 +
Mark Shannon  wrote:
> Hi everyone,
> 
> It's time for yet another PEP :)
> 
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.

On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth()
would look like?

Also, what is the `headroom` argument in
Py_CheckStackDepthWithHeadroom() supposed to represent? Bytes? Stack
frames?

Regards

Antoine.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/XG6IU6A7ZGXVMF2TXZXOZ32SIKMAHB5X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Sebastian Berg 
On Tue, 2021-01-19 at 13:31 +, Mark Shannon wrote:
> Hi everyone,
> 
> It's time for yet another PEP :)
> 
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.
> 
> Abstract
> 
> 
> This PEP proposes that machine stack overflow is treated differently 
> from runaway recursion. This would allow programs to set the maximum 
> recursion depth to fit their needs and provide additional safety
> guarantees.
> 
> The following program will run safely to completion:
> 
>  sys.setrecursionlimit(1_000_000)
> 
>  def f(n):
>  if n:
>  f(n-1)
> 
>  f(500_000)
> 
> The following program will raise a StackOverflow, without causing a
> VM 
> crash:
> 
>  sys.setrecursionlimit(1_000_000)
> 
>  class X:
>  def __add__(self, other):
>  return self + other
> 
>  X() + 1
> 


This is appreciated! I recently spend quite a bit of time trying to
solve a StackOverflow like this in NumPy (and was unable to fully
resolve it).  Of course the code triggering it was bordering on
malicious, but it would be nice if it was clear how to not segfault.

Just some questions/notes:

* We currently mostly use `Py_EnterRecursiveCall()` in situations where
we need to safe-guard against "almost python" recursions. For example
an attribute lookup that returns `self`, or a list containing itself.
In those cases the python recursion limit seems a bit nicer (lower and
easier to understand).
I am not sure it actually matters much, but my question is: Are we sure
we want to replace all (or even many) C recursion checks?

* Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering if a
new `StackOverflow` exception name is useful. It may create two names
for almost identical Python code:  If you unpack a list containing
itself compared to a mapping implementing `__getitem__` in Python you
would get different exceptions.

* `Py_CheckStackDepthWithHeadRoom()` is usually not necessary, because
`Py_CheckStackDepth()` would leave plenty of headroom for typical
clean-up?
Can we assume that DECREF's (i.e. list, tuple), will never check the
depth, so head-room is usually not necessary?  This is all good, but I
am not immediately sure when `Py_CheckStackDepthWithHeadRoom()` would
be necessary (There are probably many cases where it clearly is, but is
it ever for fairly simple code?).
What happens if the maximum stack depth is reached while a
`StackOverflow` exception is already set?  Will the current "watermark"
mechanism remain, or could there be a simple rule that an uncleared
`StackOverflow` exception ensures some additional head-room?

Cheers,

Sebastian



> ---
> 
> The full PEP can be found here:
> https://www.python.org/dev/peps/pep-0651
> 
> As always, comments are welcome.
> 
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/[email protected]/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/N456CVKWZ3E3VKPOE2DZMFLVSMOK5BSF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Mark Shannon 




On 19/01/2021 3:40 pm, Antoine Pitrou wrote:

On Tue, 19 Jan 2021 13:31:45 +
Mark Shannon  wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.


On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth()
would look like?


It would depend on the platform, but a portable-ish implementation is here:

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71



Also, what is the `headroom` argument in
Py_CheckStackDepthWithHeadroom() supposed to represent? Bytes? Stack
frames?


Bytes. I'll update the PEP.



Regards

Antoine.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/XG6IU6A7ZGXVMF2TXZXOZ32SIKMAHB5X/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/G5HQG6NMIU3XSI5TMPDBMHM623WY2YPV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Antoine Pitrou 
On Tue, 19 Jan 2021 15:54:39 +
Mark Shannon  wrote:
> On 19/01/2021 3:40 pm, Antoine Pitrou wrote:
> > On Tue, 19 Jan 2021 13:31:45 +
> > Mark Shannon  wrote:  
> >> Hi everyone,
> >>
> >> It's time for yet another PEP :)
> >>
> >> Fortunately, this one is a small one that doesn't change much.
> >> It's aim is to make the VM more robust.  
> > 
> > On the principle, no objection.
> > 
> > In practice, can you show how an implementation of Py_CheckStackDepth()
> > would look like?  
> 
> It would depend on the platform, but a portable-ish implementation is here:
> 
> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71

This doesn't tell me how `stack_limit_pointer` is computed or estimated
:-)


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/5HELIHBQATVIDWT53MJZTPFUEG5CKSOQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Mark Shannon 



On 19/01/2021 3:43 pm, Sebastian Berg wrote:

On Tue, 2021-01-19 at 13:31 +, Mark Shannon wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently
from runaway recursion. This would allow programs to set the maximum
recursion depth to fit their needs and provide additional safety
guarantees.

The following program will run safely to completion:

  sys.setrecursionlimit(1_000_000)

  def f(n):
  if n:
  f(n-1)

  f(500_000)

The following program will raise a StackOverflow, without causing a
VM
crash:

  sys.setrecursionlimit(1_000_000)

  class X:
  def __add__(self, other):
  return self + other

  X() + 1




This is appreciated! I recently spend quite a bit of time trying to
solve a StackOverflow like this in NumPy (and was unable to fully
resolve it).  Of course the code triggering it was bordering on
malicious, but it would be nice if it was clear how to not segfault.

Just some questions/notes:

* We currently mostly use `Py_EnterRecursiveCall()` in situations where
we need to safe-guard against "almost python" recursions. For example
an attribute lookup that returns `self`, or a list containing itself.
In those cases the python recursion limit seems a bit nicer (lower and
easier to understand).
I am not sure it actually matters much, but my question is: Are we sure
we want to replace all (or even many) C recursion checks?


Would it help if you had the ability to increase and decrease the 
recursion depth, as `Py_EnterRecursiveCall()` currently does?


I'm reluctant to expose it, as it might encourage C code authors to use 
it, rather than `Py_CheckStackDepth()` resulting in crashes.


To be robust, C code must make a call to `Py_CheckStackDepth()`.
To check the recursion limit as well would be extra overhead.



* Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering if a
new `StackOverflow` exception name is useful. It may create two names
for almost identical Python code:  If you unpack a list containing
itself compared to a mapping implementing `__getitem__` in Python you
would get different exceptions.


True, but they are different. One is a soft limit that can be increased, 
the other is a hard limit that cannot (at least not easily).




* `Py_CheckStackDepthWithHeadRoom()` is usually not necessary, because
`Py_CheckStackDepth()` would leave plenty of headroom for typical
clean-up?


What is "typical" clean up? I would hope that typical cleanup is to 
return immediately.



Can we assume that DECREF's (i.e. list, tuple), will never check the
depth, so head-room is usually not necessary?  This is all good, but I
am not immediately sure when `Py_CheckStackDepthWithHeadRoom()` would
be necessary (There are probably many cases where it clearly is, but is
it ever for fairly simple code?).


Ideally, Dealloc should call `Py_CheckStackDepth()`, but it will need
to be very cheap for that to be practical.

If C code is consuming the stack, its responsibility is to not overflow.
We can't make you call `Py_CheckStackDepth()`, but we can provide it, so 
you that will have no excuse for blowing the stack :)



What happens if the maximum stack depth is reached while a
`StackOverflow` exception is already set?  Will the current "watermark"
mechanism remain, or could there be a simple rule that an uncleared
`StackOverflow` exception ensures some additional head-room?


When an exception is "set", the C code should be unwinding stack,
so those states shouldn't be possible.

We can't give you extra headroom. The C stack is a fixed size.
That's why `Py_CheckStackDepthWithHeadRoom()` is provided, if 
`Py_CheckStackDepth()` fails then it is too late to do much.


Cheers,
Mark.



Cheers,

Sebastian




---

The full PEP can be found here:
https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
Code of Conduct: http://python.org/psf/codeofconduct/




___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/N456CVKWZ3E3VKPOE2DZMFLVSMOK5BSF/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Mark Shannon 




On 19/01/2021 4:15 pm, Antoine Pitrou wrote:

On Tue, 19 Jan 2021 15:54:39 +
Mark Shannon  wrote:

On 19/01/2021 3:40 pm, Antoine Pitrou wrote:

On Tue, 19 Jan 2021 13:31:45 +
Mark Shannon  wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.


On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth()
would look like?


It would depend on the platform, but a portable-ish implementation is here:

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71


This doesn't tell me how `stack_limit_pointer` is computed or estimated
:-)


It's nothing clever, and the numbers I've chosen are just off the top of 
my head.


https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Modules/_threadmodule.c#L1071






___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/5HELIHBQATVIDWT53MJZTPFUEG5CKSOQ/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/PXVS2I3UWTG2CDQUZ57IQ6NNCJ2JAT23/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Antoine Pitrou 
On Tue, 19 Jan 2021 16:28:46 +
Mark Shannon  wrote:

> On 19/01/2021 4:15 pm, Antoine Pitrou wrote:
> > On Tue, 19 Jan 2021 15:54:39 +
> > Mark Shannon  wrote:  
> >> On 19/01/2021 3:40 pm, Antoine Pitrou wrote:  
> >>> On Tue, 19 Jan 2021 13:31:45 +
> >>> Mark Shannon  wrote:  
>  Hi everyone,
> 
>  It's time for yet another PEP :)
> 
>  Fortunately, this one is a small one that doesn't change much.
>  It's aim is to make the VM more robust.  
> >>>
> >>> On the principle, no objection.
> >>>
> >>> In practice, can you show how an implementation of Py_CheckStackDepth()
> >>> would look like?  
> >>
> >> It would depend on the platform, but a portable-ish implementation is here:
> >>
> >> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71
> >>   
> > 
> > This doesn't tell me how `stack_limit_pointer` is computed or estimated
> > :-)  
> 
> It's nothing clever, and the numbers I've chosen are just off the top of 
> my head.
> 
> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Modules/_threadmodule.c#L1071

What about the main thread?

Is `stack_limit_pointer` some kind of thread-local?  I suppose nothing
stops an OS to use different virtual addresses for the stacks of
different threads (especially if randomization of stack addresses is
desired).

Regards

Antoine.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/SKJMONH55RMKJNZVI4K7AVOIUOK77WZ7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please explain how to migrate when a function is removed, thanks ;-)

2021-01-19 Thread Victor Stinner 
A PR was proposed to document the removal of the loop parameter:
https://github.com/python/cpython/pull/24256

Victor

On Tue, Jan 19, 2021 at 1:34 PM Victor Stinner  wrote:
>
> Hi,
>
> We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
> are facing many Python 3.10 incompatible changes. Most changes were
> planned with a deprecation period, but, as usual, projects are not
> tested with DeprecationWarning treated as errors, and so tons of
> packages now fail to build.
>
> I'm not here to talk about DeprecationWarning, but how we communicate
> on incompatible changes made on purpose. For example, What's New in
> Python 3.8 announces: "In asyncio, the explicit passing of a loop
> argument has been deprecated and will be removed in version 3.10". As
> expected, the parameter was removed in Python 3.10 (bpo-42392), but I
> cannot see anything in What's New in Python 3.10. The problem is that
> I don't know how to fix projects broken by this change. I'm not
> complaining about this specific change, but more generally.
>
> I strongly suggest to well document incompatible changes. The bare
> minimum is to mention them in "Porting to Python 3.10" section:
> https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10
>
> A link to the bpo sometimes helps to understand how to port code. But
> I would really appreciate if authors of incompatible changes would
> explain how to add Python 3.10 support to existing projects, without
> losing support for older Python versions. Not just "this function is
> now removed, good luck!" :-)
>
> I didn't touch asyncio for at least 1 year, so I don't know what
> happens if I remove a loop argument. Does an application remain
> compatible with Python 3.6 without passing loop?
>
> I know that I made multiple incompatible changes myself and I'm sure
> that the documentation should probably be enhanced, but I also expect
> others to help on that! ;-)
>
> Please think about people who have to port 4000 Python projects to Python 
> 3.10!
>
> ---
>
> The best would be to ship a tool which adds Python 3.10 support to
> existing projects without losing support with Python 3.6. Maybe
> something like https://github.com/asottile/pyupgrade could be used for
> that? pyupgrade looks more specific to the Python syntax, than the
> usage of the stdlib.
>
> I wrote such tool to add Python 3.10 support to C extensions without
> losing support with Python 2.7. It relies on a header file
> (pythoncapi_compat.h) which provides new C API functions on old Python
> versions.
> https://github.com/pythoncapi/pythoncapi_compat
>
> For example, it replaces "obj->ob_refcnt = refcnt;" with
> "Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
> 3.8 and older (function added to Python 3.9)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/CVLDV7VIVENV6HMQ4PRAN3VRFU26CMJI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Guido van Rossum 
I'm not clear on how you plan to implement this in CPython.

I can totally see that if a Python function calls another Python function,
you can avoid the C stack frame and hence you can have as many Python call
levels as you want.

However, there are many scenarios where a Python function calls a C
function (e.g. `filter()`, or `dict.__setitem__()`) and that C function at
some point calls a Python function (e.g. the `__hash__()` method of the
key, or even the `__del__()` method of the value being replaced). Then that
Python function can recursively do a similar thing.

Are you proposing to also support that kind of thing to go on for a million
levels of C stack frames?

(Do we even have a cross-platform way of avoiding segfaults due to C stack
overflow?)

On Tue, Jan 19, 2021 at 5:38 AM Mark Shannon  wrote:

> Hi everyone,
>
> It's time for yet another PEP :)
>
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.
>
> Abstract
> 
>
> This PEP proposes that machine stack overflow is treated differently
> from runaway recursion. This would allow programs to set the maximum
> recursion depth to fit their needs and provide additional safety
> guarantees.
>
> The following program will run safely to completion:
>
>  sys.setrecursionlimit(1_000_000)
>
>  def f(n):
>  if n:
>  f(n-1)
>
>  f(500_000)
>
> The following program will raise a StackOverflow, without causing a VM
> crash:
>
>  sys.setrecursionlimit(1_000_000)
>
>  class X:
>  def __add__(self, other):
>  return self + other
>
>  X() + 1
>
> ---
>
> The full PEP can be found here:
> https://www.python.org/dev/peps/pep-0651
>
> As always, comments are welcome.
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/KQLHJY3CDWK42T5GTCB6QYIH4J3TGNCT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Sebastian Berg 
On Tue, 2021-01-19 at 16:22 +, Mark Shannon wrote:
> 
> 
> On 19/01/2021 3:43 pm, Sebastian Berg wrote:
> > On Tue, 2021-01-19 at 13:31 +, Mark Shannon wrote:
> > > Hi everyone,
> > > 
> > > It's time for yet another PEP :)
> > > 
> > > Fortunately, this one is a small one that doesn't change much.
> > > It's aim is to make the VM more robust.
> > > 
> > > Abstract
> > > 
> > > 
> > > This PEP proposes that machine stack overflow is treated
> > > differently
> > > from runaway recursion. This would allow programs to set the
> > > maximum
> > > recursion depth to fit their needs and provide additional safety
> > > guarantees.
> > > 
> > > The following program will run safely to completion:
> > > 
> > >   sys.setrecursionlimit(1_000_000)
> > > 
> > >   def f(n):
> > >   if n:
> > >   f(n-1)
> > > 
> > >   f(500_000)
> > > 
> > > The following program will raise a StackOverflow, without causing
> > > a
> > > VM
> > > crash:
> > > 
> > >   sys.setrecursionlimit(1_000_000)
> > > 
> > >   class X:
> > >   def __add__(self, other):
> > >   return self + other
> > > 
> > >   X() + 1
> > > 
> > 
> > 
> > This is appreciated! I recently spend quite a bit of time trying to
> > solve a StackOverflow like this in NumPy (and was unable to fully
> > resolve it).  Of course the code triggering it was bordering on
> > malicious, but it would be nice if it was clear how to not
> > segfault.
> > 
> > Just some questions/notes:
> > 
> > * We currently mostly use `Py_EnterRecursiveCall()` in situations
> > where
> > we need to safe-guard against "almost python" recursions. For
> > example
> > an attribute lookup that returns `self`, or a list containing
> > itself.
> > In those cases the python recursion limit seems a bit nicer (lower
> > and
> > easier to understand).
> > I am not sure it actually matters much, but my question is: Are we
> > sure
> > we want to replace all (or even many) C recursion checks?
> 
> Would it help if you had the ability to increase and decrease the 
> recursion depth, as `Py_EnterRecursiveCall()` currently does?
> 
> I'm reluctant to expose it, as it might encourage C code authors to
> use 
> it, rather than `Py_CheckStackDepth()` resulting in crashes.
> 
> To be robust, C code must make a call to `Py_CheckStackDepth()`.
> To check the recursion limit as well would be extra overhead.
> 
> > 
> > * Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering
> > if a
> > new `StackOverflow` exception name is useful. It may create two
> > names
> > for almost identical Python code:  If you unpack a list containing
> > itself compared to a mapping implementing `__getitem__` in Python
> > you
> > would get different exceptions.
> 
> True, but they are different. One is a soft limit that can be
> increased, 
> the other is a hard limit that cannot (at least not easily).


Right. I think my confusion completely resolves around your proposed
change of `Py_EnterRecursiveCall()`.

A simple example (written in C):

def depth(obj, current=0):
Py_EnterRecursiveCall()

if isinstance(obj, sequence):  # has the sequence slots
return depth(obj[0], current+1)
return current

will never hit the "depth" limit for a self containing list or even
sequence (as long as `GetItem` can use the C-level slot).

But `obj[0]` could nevertheless return a non-trivial object (one with
`__del__`, definitely a container with unrelated objects that could use
deleting).

As the author of the function, I have no knowledge over how much stack
space cleaning those up may require?
And say someone adds a check for `Py_CheckStackDepth()` inside a
dealloc, then this might have to cause a fatal error?

Maybe it should even be a fatal error by default in some cases?

Also, if the code is slow, the previous recursion may guard against
hanging (arguably, if that is the case I probably add an interrupt
check, I admit).


Long story short, I will trust you guys on it of course, but I am not
yet convinced that replacing the check will actually do any good (as
opposed to adding and/or providing the additional check) or even be a
service to users (since I assume that the vast majority do not crank up
the recursion limit to huge values).

Cheers,

Sebastian



> 
> > 
> > * `Py_CheckStackDepthWithHeadRoom()` is usually not necessary,
> > because
> > `Py_CheckStackDepth()` would leave plenty of headroom for typical
> > clean-up?
> 
> What is "typical" clean up? I would hope that typical cleanup is to 
> return immediately.
> 
> > Can we assume that DECREF's (i.e. list, tuple), will never check
> > the
> > depth, so head-room is usually not necessary?  This is all good,
> > but I
> > am not immediately sure when `Py_CheckStackDepthWithHeadRoom()`
> > would
> > be necessary (There are probably many cases where it clearly is,
> > but is
> > it ever for fairly simple code?).
> 
> Ideally, Dealloc should call `Py_CheckStackDepth()`, but it will need
> to

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Mark Shannon 



On 19/01/2021 5:48 pm, Guido van Rossum wrote:

I'm not clear on how you plan to implement this in CPython.

I can totally see that if a Python function calls another Python 
function, you can avoid the C stack frame and hence you can have as many 
Python call levels as you want.


However, there are many scenarios where a Python function calls a C 
function (e.g. `filter()`, or `dict.__setitem__()`) and that C function 
at some point calls a Python function (e.g. the `__hash__()` method of 
the key, or even the `__del__()` method of the value being replaced). 
Then that Python function can recursively do a similar thing.


Indeed, that is the second case below, where a Python __add__
function recursively performs addition. Most likely, the C stack will 
get exhausted before the recursion limit is hit, so you'll get a 
StackOverflow exception.




Are you proposing to also support that kind of thing to go on for a 
million levels of C stack frames?


No, most likely 10k to 20k calls before a StackOverflow exception.



(Do we even have a cross-platform way of avoiding segfaults due to C 
stack overflow?)


Arithmetic and comparisons on pointers from within the C stack may not 
strictly conform to the C standard, but it works just fine.
It's pretty standard VM implementation stuff. All the JVMs do this sort 
of thing.


IMO practically beats purity in this case, and crashing less has to be a 
good thing.



A rather hacky proof of concept for the stack overflow handling is here:

https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation


Cheers,
Mark.



On Tue, Jan 19, 2021 at 5:38 AM Mark Shannon > wrote:


Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently
from runaway recursion. This would allow programs to set the maximum
recursion depth to fit their needs and provide additional safety
guarantees.

The following program will run safely to completion:

      sys.setrecursionlimit(1_000_000)

      def f(n):
          if n:
              f(n-1)

      f(500_000)

The following program will raise a StackOverflow, without causing a VM
crash:

      sys.setrecursionlimit(1_000_000)

      class X:
          def __add__(self, other):
              return self + other

      X() + 1

---

The full PEP can be found here:
https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- [email protected]

To unsubscribe send an email to [email protected]

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/[email protected]/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
Code of Conduct: http://python.org/psf/codeofconduct/



--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/PYJHVAA63AWOL72OJMNFDY7VODIT5KM7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Guido van Rossum 
On Tue, Jan 19, 2021 at 10:08 AM Mark Shannon  wrote:

>
>
> On 19/01/2021 5:48 pm, Guido van Rossum wrote:
> > I'm not clear on how you plan to implement this in CPython.
> >
> > I can totally see that if a Python function calls another Python
> > function, you can avoid the C stack frame and hence you can have as many
> > Python call levels as you want.
> >
> > However, there are many scenarios where a Python function calls a C
> > function (e.g. `filter()`, or `dict.__setitem__()`) and that C function
> > at some point calls a Python function (e.g. the `__hash__()` method of
> > the key, or even the `__del__()` method of the value being replaced).
> > Then that Python function can recursively do a similar thing.
>
> Indeed, that is the second case below, where a Python __add__
> function recursively performs addition. Most likely, the C stack will
> get exhausted before the recursion limit is hit, so you'll get a
> StackOverflow exception.
>
> >
> > Are you proposing to also support that kind of thing to go on for a
> > million levels of C stack frames?
>
> No, most likely 10k to 20k calls before a StackOverflow exception.
>

Okay, that helps my understanding of the proposal. So StackOverflow really
means "C stack overflow"? And the plan seems to be

- no intervening C frame for Python-to-Python calls
- fix all C code to explicitly check for stack overflow (even in `__del__`)
- decouple RecursionError (Python) from StackOverflow (C)

If you can pull that off I am all for it.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/5K5VZZS7J7ZSDLKEC6NKGSMLOPL3S5C2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Pablo Galindo Salgado 
Hi Mark,

Thanks for gathering this proposal! Looks very interesting. I have some 
preliminary questions: how is this going to affect
the "py-bt" command of the gdb helpers 
(https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
and other similar tools that produce a unified Python-C backtrace? There are 
several debuggers that
rely on the fact that they can merge the C stack and the Python stack by 
substituting every call to "_PyEval_EvalFrameDefault"
and friends for the respective Python frame obtained separately or by 
inspecting the frame object in the stack (like gdb does).

Is this change going to affect these tools or they will continue working as 
before? In case this change will affect this tools,
is there any workaround to produce the unified C/Python call stack given the 
Python stack and the C stack?

Kind regards,
Pablo Galindo Salgado
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

2021-01-19 Thread Terry Reedy 

On 1/19/2021 10:01 AM, Mark Shannon wrote:


The following program will run safely to completion:


I interpreted this to mean 'works now', on whatever system you tested 
this on.  You question suggests that you meant "fails now but will work 
with a successful patch for the PEP".



 sys.setrecursionlimit(1_000_000)


On Windows, this recursion limit increase lets the function run about 
2160 loops (when run with IDLE) instead of 1000, but makes the behavior 
WORSE by replacing the exception with a silent failure.  This could be 
considered worse than a crash.



 def f(n):
 if n:
 f(n-1)

 f(500_000)


I'm not sure whether you are saying that this doesn't work now, that it 
can't work, or that it shouldn't work.



If that it doesn't work now, then I agree.


On my Win 10, setting the recursion limit to 2160 (2135 in IDLE, which 
bumps it up to account for additional stack space used by IDLE) results 
in RecursionError.  Much higher and the failure is silent.  Always 
getting a RecursionError would itself be an improvement.


So I am saying that current recursion limit handling on Windows is 
terrible and perhaps worse than on Linux.


--
Terry Jan Reedy

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/XSPGPFHIP5VWDBEWQHKSU4A55DHFWAGF/
Code of Conduct: http://python.org/psf/codeofconduct/