[Python-Dev] Re: Adding a scarier warning to object.__del__?

2020-01-03 Thread Petr Viktorin 
Could there be a clearer separation between the language semantics and 
current CPython behavior?


On 2020-01-02 03:53, Yonatan Zunger wrote:

Oh, I'm absolutely thinking about clarity. Something like:



This method is called when the instance is about to be destroyed. 
Because it may be called by either the ordinary Python execution flow or 
by the garbage collector, it has very unusual semantics, and must be 
treated with great care.



The precise semantics of when __del__ is called on an object are 
implementation-dependent. For example:
* It might be invoked during the normal interpreter flow at a moment 
like function return, if the reference count for an object has dropped 
to zero;
* It might be invoked by the garbage collector, which happens during the 
normal interpreter flow but is called from an arbitrary thread;
* It might be invoked during interpreter shutdown, after various global 
variables (including modules!) have already been deleted, or other 
threads' interpreters have already stopped.
* It may not be invoked at all; it is /not guaranteed/ that __del__() 
methods are called for objects that still exist when the interpreter exits.


Note that similar caveats apply for alternatives to __del__: 
weakref.finalize and atexit.register.


Doing any of the following in an implementation of __del__() may easily 
lead to bugs, nondeterministic behavior, and inconsistent behavior 
across different implementations or versions of Python:


* Assuming that any variables outside self exist, including 
modules or global variables;
* Attempting to acquire locks or otherwise block, since they may 
(e.g.) be invoked inside a thread which already holds those resources;
* Relying on being able to signal other threads and wait for their 
responses (even thread joins), since those threads' interpreters may 
have exited;
* Causing failures (e.g. leaving external devices in an invalid 
state) if __del__ is /never/ invoked.


In particular, del x does not call x.__del__() directly: the call may be 
delayed or not happen at all.


In CPython, del x decrements the reference count for x by one, and 
x.__del__() is only called either when x’s reference count reaches zero, 
or by the cyclic garbage collector at an unspecified time (e.g. if a 
reference cycle prevents the reference count from going to zero).


If a base class has a __del__() method, the derived class’s __del__() 
method, if any, must explicitly call it to ensure proper deletion of the 
base class part of the instance.


It is possible (though not recommended!) for the __del__() method to 
postpone destruction of the instance by creating a new reference to it. 
This is called object resurrection. It is implementation-dependent 
whether __del__() is called a second time when a resurrected object is 
about to be destroyed.
The current CPython implementation usually calls __del__ only once for 
each object, but this is not guaranteed in all cases.




On Wed, Jan 1, 2020 at 12:57 PM Andrew Svetlov > wrote:


If the warning text tells about the delayed execution -- I'm +1.
-1 for something like "just don't use __del__, it is dangerous".

On Wed, Jan 1, 2020, 21:51 Gregory P. Smith mailto:[email protected]>> wrote:



On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov
mailto:[email protected]>> wrote:

__del__ is very useful not for interpreter shutdown but as a
regular
destructor in object's lifecycle.


The reason we should warn people against ever implementing
__del__ is that people rarely actually understand object
lifecycle.  Its presence guarantees delayed garbage collection
and that its code will code execute in a context way outside of
normal control flow that all other methods are invoked from.

-gps


Action on the shutdown is another beast.
Personally, I prefer to do all finalization works by
explicit calls instead.

On Wed, Jan 1, 2020 at 2:39 AM Yonatan Zunger
mailto:[email protected]>> wrote:
 >
 > Hey everyone,
 >
 > I just encountered yet another reason to beware of
__del__: when it's called during interpreter shutdown, for
reasons which are kind of obvious in retrospect, if it calls
notify() on a threading.Condition, the waiting thread may or
may not ever actually receive it, and so if it does that and
then tries to join() the thread the interpreter may hang in
a hard-to-debug way.
 >
 > This isn't something that can reasonably be fixed, and
(like in most cases involving __del__) there's a very simple
fix of using weakref.finalize instead. My question for the
dev list: How would people feel about changing the
documentation for the method to more

[Python-Dev] Re: Adding a scarier warning to object.__del__?

2020-01-03 Thread Jeff Allen 

On 02/01/2020 02:53, Yonatan Zunger wrote:

Oh, I'm absolutely thinking about clarity. ...

Could any revision also be clear what is *required of Python the 
language* vs. what is a CPython implementation detail? I always 
appreciate this care. There is good practice here and elsewhere in the 
existing documentation, but drift is easy for those steeped in CPython 
implementation. In the present case, it's a matter of avoiding an 
explicit requirement for a reference-counting approach to lifecycle 
management. Here the places in the proposal a change could achieve that.




The precise semantics of when __del__ is called on an object are 
implementation-dependent. For example:
* It might be invoked during the normal interpreter flow at a moment 
like function return, ...


We should continue here "... immediately the object is no nonger 
referenced;"


(It might not be called immediately, but that's implied by your 
implementation-dependent "might be".)



Note that del x doesn’t directly call x.__del__() — the former 
decrements the reference count for x by one, and the latter is only 
called when x’s reference count reaches zero. Depending on the 
implementation, it is possible for a reference cycle to prevent the 
reference count of an object from going to zero. (e.g., in CPython, a 
common cause of reference cycles is when an exception is caught and 
stored in a local variable; the exception contains a reference to the 
traceback, which in turn references the locals of all frames caught in 
the traceback.) In this case, the cycle will be later detected and 
deleted by the cyclic garbage collector.


I realise that most of this paragraph is existing text rearranged, and 
currently it fails to make the distinction I'm looking for in the "note" 
part. But it is clear in the next paragraph. I think it better to say, 
closer to the current text:


"""Note:: ``del x`` does not call ``x.__del__()`` directly. After ``del 
x``, variable ``x`` is undefined (or unbound). If the object it 
referenced is now no longer referenced at all,  that object's 
``__del__()`` might be called, immediately or later, subject to the 
caveats already given.


*CPython implementation detail:* ``del x`` decrements the reference 
count of the object by one, and if that makes it zero, ``x.__del__()`` 
will be called immediately. It is possible for a reference cycle to 
prevent the reference count of any object in it from going to zero. A 
common cause of reference cycles is when an exception is caught and 
stored in a local variable; the exception contains a reference to the 
traceback, which in turn references the locals of all frames caught in 
the traceback. In this case, the cycle will be detected later and its 
objects deleted by the cyclic garbage collector."""



If a base class has a __del__() method, the derived class’s __del__() 
method, if any, must explicitly call it to ensure proper deletion of 
the base class part of the instance.


Possibly this thought belings with the "implementations of __del__(): * 
Must ..." paragraph.


But also, while I think there is scope for a better guidance, this is 
getting a bit long. Should there be a "HOW TO write a __del__ method 
(and how to avoid it)" to contain the advisory points being made? 
In-lining advice here, on how to survive the infernal circles of 
__del__, dilutes the scariness of the warning not to enter at all.


---

Jeff Allen

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/A32QEUP4R6XTY5LQW56LKWJ3XBUZCHOR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python Documentation and AIX specifics - how to proceed?

2020-01-03 Thread Petr Viktorin 

On 2019-12-26 14:35, Michael wrote:

First - best wishes all for a happy and healthy 2020!

As my nickname implies - my primary means to contribute to Python is
with regard to AIX. One of the things I recently came across is
Misc/README.AIX which was last updated sometime between 2010 and 2014. I
am thinking a facelift is in order. Assuming 2010-2011 as the original
date - much has changed and many of the comments are no longer accurate.

Before saying so, I would like to check here that - having all tests
pass on the 3.8 bot implies that there are no outstanding issues. As to
the historical issues in the current document - these could either be
deleted, or a short text describing when they were resolved (e.g.,
Python 3.7, or just resolved, but noone knows exactly how or when -
whether it was a Python change, or a platform (AIX) change.
Delete them. The README's history is in Git. If you know additional 
details (like when an issue was resolved), you can say that in the 
commit message.


What I see as being more relevant is the description re: how to build
Python for AIX - for the "do it youself"-ers.
So, besides the direct question re: what to say about the old "known
issues" and whether there are no known issues aka, no issues identified
via the standard tests - I would appreciate feedback on what is
considered appropriate - anno 2020 - for any Misc/README.platform text.

Additionally, I am willing to work on other areas of the standard
documentation where it is either needed or considered appropriate for
platform specific details and/or examples.

This is not something I would try to get done in a single PR, Instead I
am thinking a single -longer term- issue - and multiple PR's to work
through corrections and additions during 2020. Focus on Python 3.9 and
beyond yet where appropriate backlevel to Python 3.8 or even 3.7.

Sincerely,

Michael



Some devs have been setting GitHub repositories for larger projects, 
with issues (TODO lists), collaboration, wikis or even blog-like 
documentation. For example:


https://github.com/ericsnowcurrently/multi-core-python
https://faster-cpython.readthedocs.io/
https://github.com/python/typing
https://github.com/python/asyncio

It helps to have a space where you can focus on a task without having to 
prove that "this is good for CPython right now" *all* the time.


You could replace README.AIX text with a link there. Afterwards, I think 
the devguide or Python documentation could link to such a space for the 
AIX build instructions; then, when finalized, the instructions 
themselves could be in Python docs or in a new README.AIX. But I do 
recommend to separate writing the text from proposing it for inclusion 
in CPython.


___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/Y4I7CDFIRD7H64THTMHWWHIJ7GCXEWSH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Last call for comments on PEP 573 (Module State Access from C Extension Methods)

2020-01-03 Thread Petr Viktorin 

On 2019-11-25 13:15, Stefan Behnel wrote:

Hi all,

I think that PEP 573 is ready to be accepted, to greatly improve the state
of extension modules in CPython 3.9.

https://www.python.org/dev/peps/pep-0573/

It has come a long way since the original proposal and went through several
iterations and discussions by various interested people, effectively
reducing its scope quite a bit. So this is the last call for comments on
the latest version of the PEP, before I will pronounce on it. Please keep
the discussion in this thread.


I have received substantial private feedback, and I'll need to make 
another update to:


## Better separate C-API additions:

- PyType_FromModuleAndSpec
- PyType_GetModule
- PyType_DefiningTypeFromSlotFunc
- PyType_GetModuleState

- METH_METHOD flag
- PyCMethod (function signature)

from CPython implementation details:

- ht_module (member of the undocumented PyHeapTypeObject)
- PyCMethodObject (extends the undocumented  PyCFunctionObject; used 
internally to hold information to pass to PyCMethod)

- PyCFunction_GET_CLASS (helper for the internal class)
- defining_class clinic converter

(See 
https://mail.python.org/archives/list/[email protected]/message/B2VDVLABM4RQ4ATEJXFZYWEGTBZPUBKW/ 
for a proposal to formalize this distinction as "rings", and for some 
reasons why it is good.)


## Clarify that ht_module is not inherited

## Specify what "defining class" means more rigorously
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/336BSKX6NTW6FBB5JCFZOHVBI2PUF2MW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Summary of Python tracker Issues

2020-01-03 Thread Python tracker 


ACTIVITY SUMMARY (2019-12-27 - 2020-01-03)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7227 (+24)
  closed 43726 (+38)
  total  50953 (+62)

Open issues with patches: 2842 


Issues opened (48)
==

#27973: urllib.urlretrieve() fails on second ftp transfer
https://bugs.python.org/issue27973  reopened by pablogsal

#39114: Python 3.9.0a2 changed how finally/return is traced
https://bugs.python.org/issue39114  reopened by nedbat

#39141: IDLE Clear function returns 256 on Mac OS Catalina
https://bugs.python.org/issue39141  opened by [email protected]

#39143: Implementing sub-generation steps in the gc
https://bugs.python.org/issue39143  opened by pablogsal

#39146: too much memory consumption in re.compile unicode
https://bugs.python.org/issue39146  opened by Zhipeng Xie

#39147: using zipfile with root privilege shows FileNotFoundError
https://bugs.python.org/issue39147  opened by Patrick Liu

#39148: DatagramProtocol + IPv6 does not work with ProactorEventLoop
https://bugs.python.org/issue39148  opened by alex.gronholm

#39150: See if PyToken_OneChar would be faster as a lookup table
https://bugs.python.org/issue39150  opened by petdance

#39151: Simplify the deep-first-search of the assembler
https://bugs.python.org/issue39151  opened by pablogsal

#39152: Faulty override of tkinter.Misc.configure in tkinter.ttk.Scale
https://bugs.python.org/issue39152  opened by glombardo

#39153: Clarify refcounting semantics of PyDict_SetItem[String]
https://bugs.python.org/issue39153  opened by ncoghlan

#39154: "utf8" not always a synonym for "utf-8" in lib2to3
https://bugs.python.org/issue39154  opened by Peter Ludemann

#39155: "utf8-sig" missing from codecs (inconsistency)
https://bugs.python.org/issue39155  opened by Peter Ludemann

#39156: Break up COMPARE_OP into logically distinct operations.
https://bugs.python.org/issue39156  opened by Mark.Shannon

#39157: test_pidfd_send_signal can fail on some systems with Permissio
https://bugs.python.org/issue39157  opened by pablogsal

#39159: Ideas for making ast.literal_eval() usable
https://bugs.python.org/issue39159  opened by rhettinger

#39160: ./configure --help has inconsistencies in style
https://bugs.python.org/issue39160  opened by anthony shaw

#39161: Py_NewInterpreter docs need updating for multi-phase initializ
https://bugs.python.org/issue39161  opened by ncoghlan

#39162: setup.py not picking up tkinter headers
https://bugs.python.org/issue39162  opened by anthonypjshaw

#39164: PyErr_GetExcInfo does not allow to retrieve for an arbitrary t
https://bugs.python.org/issue39164  opened by jd

#39165: Completeness and symmetry in RE, avoid `findall(...)[0]`
https://bugs.python.org/issue39165  opened by apalala

#39166: Python 3.9.0a2 changed how "async for" traces its final iterat
https://bugs.python.org/issue39166  opened by nedbat

#39168: Generic type subscription is a huge toll on Python performance
https://bugs.python.org/issue39168  opened by Ruslan Dautkhanov

#39169: TypeError: 'int' object is not callable if the signal handler 
https://bugs.python.org/issue39169  opened by Ronald Li

#39170: Sqlite3 row_factory for attribute access: NamedRow
https://bugs.python.org/issue39170  opened by jidn

#39171: Missing default root in tkinter simpledialog.py
https://bugs.python.org/issue39171  opened by dominic108

#39173: _AttributeHolder of argparse should support the sort function 
https://bugs.python.org/issue39173  opened by shihai1991

#39174: unicodedata.normalize failing with NFD and NFKD for some chara
https://bugs.python.org/issue39174  opened by Lee Collins

#39177: In tkinter, simple dialogs, askstrings, etc. with flexible coo
https://bugs.python.org/issue39177  opened by dominic108

#39179: pandas tz_convert() seems to report incorrect date conversion
https://bugs.python.org/issue39179  opened by Seeking.that

#39180: Missing getlines func documentation from linecache module
https://bugs.python.org/issue39180  opened by khalidmammadov

#39181: Add `os.makedirs()` as `Path.mkdir()` equivalent in correspond
https://bugs.python.org/issue39181  opened by jack1142

#39182: sys.addaudithook(hook) loops indefinitely on mismatch for hook
https://bugs.python.org/issue39182  opened by Dutcho

#39184: Many command execution functions are not raising auditing even
https://bugs.python.org/issue39184  opened by Saiyang Gou

#39185: Add quiet and detailed verbosity levels to build.bat
https://bugs.python.org/issue39185  opened by anthonypjshaw

#39187: urllib.robotparser does not respect the longest match for the 
https://bugs.python.org/issue39187  opened by gallicrooster

#39188: recent change when passing a Union to a function
https://bugs.python.org/issue39188  opened by Kevin Schlossser

#39189: Use io.DEFAULT_BUFFER_SIZE for filecmp BUFSIZE variable
https://bugs.python.org/issue39189  opened by

[Python-Dev] Re: Adding a scarier warning to object.__del__?

2020-01-03 Thread Yonatan Zunger 
Jeff, I like all of your proposals. It does get a bit long, but I think
it's not so long that it feels out-of-place in the data model docs, which
is where you expect weird stuff to live. (I mean, look how long the
discussion of metaclasses is -- and I certainly wouldn't want to eliminate
that!)

On Fri, Jan 3, 2020 at 3:56 AM Jeff Allen  wrote:

> On 02/01/2020 02:53, Yonatan Zunger wrote:
>
> Oh, I'm absolutely thinking about clarity. ...
>
> Could any revision also be clear what is *required of Python the language*
> vs. what is a CPython implementation detail? I always appreciate this care.
> There is good practice here and elsewhere in the existing documentation,
> but drift is easy for those steeped in CPython implementation. In the
> present case, it's a matter of avoiding an explicit requirement for a
> reference-counting approach to lifecycle management. Here the places in the
> proposal a change could achieve that.
>
>
> The precise semantics of when __del__ is called on an object are
> implementation-dependent. For example:
> * It might be invoked during the normal interpreter flow at a moment like
> function return, ...
>
> We should continue here "... immediately the object is no nonger
> referenced;"
>
> (It might not be called immediately, but that's implied by your
> implementation-dependent "might be".)
>
> Note that del x doesn’t directly call x.__del__() — the former decrements
> the reference count for x by one, and the latter is only called when x’s
> reference count reaches zero. Depending on the implementation, it is
> possible for a reference cycle to prevent the reference count of an object
> from going to zero. (e.g., in CPython, a common cause of reference cycles
> is when an exception is caught and stored in a local variable; the
> exception contains a reference to the traceback, which in turn references
> the locals of all frames caught in the traceback.) In this case, the cycle
> will be later detected and deleted by the cyclic garbage collector.
>
> I realise that most of this paragraph is existing text rearranged, and
> currently it fails to make the distinction I'm looking for in the "note"
> part. But it is clear in the next paragraph. I think it better to say,
> closer to the current text:
>
> """Note:: ``del x`` does not call ``x.__del__()`` directly. After ``del
> x``, variable ``x`` is undefined (or unbound). If the object it referenced
> is now no longer referenced at all,  that object's ``__del__()`` might be
> called, immediately or later, subject to the caveats already given.
>
> *CPython implementation detail:* ``del x`` decrements the reference count
> of the object by one, and if that makes it zero, ``x.__del__()`` will be
> called immediately. It is possible for a reference cycle to prevent the
> reference count of any object in it from going to zero. A common cause of
> reference cycles is when an exception is caught and stored in a local
> variable; the exception contains a reference to the traceback, which in
> turn references the locals of all frames caught in the traceback. In this
> case, the cycle will be detected later and its objects deleted by the
> cyclic garbage collector."""
>
>
> If a base class has a __del__() method, the derived class’s __del__()
> method, if any, must explicitly call it to ensure proper deletion of the
> base class part of the instance.
>
> Possibly this thought belings with the "implementations of __del__(): *
> Must ..." paragraph.
>
> But also, while I think there is scope for a better guidance, this is
> getting a bit long. Should there be a "HOW TO write a __del__ method (and
> how to avoid it)" to contain the advisory points being made? In-lining
> advice here, on how to survive the infernal circles of __del__, dilutes the
> scariness of the warning not to enter at all.
>
> ---
>
> Jeff Allen
>
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/A32QEUP4R6XTY5LQW56LKWJ3XBUZCHOR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/5FO4CRYFPJGVZJDYYXMUQBTLVBZ5DMVN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Adding a scarier warning to object.__del__?

2020-01-03 Thread Inada Naoki 
I don't want to recommend atexit or weakref because they have their pitfall.

I think the context manager should be considered at first when people want
to use __del__.

On Wed, Jan 1, 2020 at 9:35 AM Yonatan Zunger  wrote:
>
> Hey everyone,
>
> I just encountered yet another reason to beware of __del__: when it's called 
> during interpreter shutdown, for reasons which are kind of obvious in 
> retrospect, if it calls notify() on a threading.Condition, the waiting thread 
> may or may not ever actually receive it, and so if it does that and then 
> tries to join() the thread the interpreter may hang in a hard-to-debug way.
>
> This isn't something that can reasonably be fixed, and (like in most cases 
> involving __del__) there's a very simple fix of using weakref.finalize 
> instead. My question for the dev list: How would people feel about changing 
> the documentation for the method to more bluntly warn people against using 
> it, and refer them to weakref.finalize and/or atexit.register as an 
> alternative? The text already has an undertone of "lasciate ogni speranza, 
> voi ch'entrate" but it may be helpful to be more explicit to avoid people 
> getting caught in unexpected pitfalls.
>
> Yonatan
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/[email protected]/message/AAZQBWD6PHC4PVNCCPX4A2745SS7B3LS/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Inada Naoki  
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/RSG6DHNG7ZTJCNYSAMVOFFQSD7HFSGKE/
Code of Conduct: http://python.org/psf/codeofconduct/