STINNER Victor <vstin...@python.org> added the comment:
Right now, threading_shutdown_interrupted.py does crash on the master branch
(commit 05e4a296ecc127641160a04f39cc02c0f66a8c27). I reintroduced the bug with:
commit 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6
Author: Victor Stinner <vstin...@python.org>
Date: Mon Mar 9 23:37:49 2020 +0100
bpo-19466: Py_Finalize() clears daemon threads earlier (GH-18848)
Clear the frames of daemon threads earlier during the Python shutdown to
call objects destructors. So "unclosed file" resource warnings are now
emitted for daemon threads in a more reliable way.
I investigated this bug with Pablo Galindo Salgado. When the bug triggers,
there are 2 strong references to a Frame object:
* Traceback object
* Generator object
The bug occurs in "except ValueError as exc:" of EvilThread.coroutine().
The exception contains a traceback object which has a strong reference to the
frame.
The frame object contains "exc" variable (the exception) which contains a
reference to the tracecback (exc.__traceback__) which contains a refererence to
the frame (tb.tb_frame): there is a reference cycle. (It's not really an issue
by itself).
There are 2 strong references to the Frame object.
Py_FinalizeEx() calls _PyThreadState_DeleteExcept() which doesn't clear the
reference cycle. But PyThreadState_Clear() calls Py_CLEAR(tstate->frame) which
reduces the Frame reference counter from 2 to 1, even if there are still 2
strong references to it (traceback and generator).
tstate->frame is only set at 3 places:
* new_threadstate(): when a Python thread state is created (it's initialized to
NULL)
* _PyEval_EvalFrameDefault() entry: set to frame
* _PyEval_EvalFrameDefault() exit: set to frame->f_back (which can be NULL
By the way, _PyEval_EvalFrameDefault() has a bug: tstate->frame is not reset if
_PyCode_InitOpcache() fails: bpo-40048.
Py_FinalizeEx() calls _PyRuntimeState_SetFinalizing(runtime, tstate) and so
threads which tries to take the GIL must exit immediately. Extract of
take_gil():
---
static inline int
tstate_must_exit(PyThreadState *tstate)
{
PyThreadState *finalizing = _PyRuntimeState_GetFinalizing(&_PyRuntime);
return (finalizing != NULL && finalizing != tstate);
}
static void
take_gil(PyThreadState *tstate)
{
...
if (tstate_must_exit(tstate)) {
PyThread_exit_thread();
}
...
}
---
The thread is exited in the middle of _PyEval_EvalFrameDefault() by take_gil()
which calls PyThread_exit_thread() because the thread must exit (because the
main thread destroyed its Python thread state and so accessing it would crash
the thread).
_PyThreadState_DeleteExcept() calls PyThreadState_Clear() which calls
Py_CLEAR(tstate->frame): this case only occurs because the thread exited in the
middle of _PyEval_EvalFrameDefault(). In the normal case,
_PyEval_EvalFrameDefault() always resets tstate->frame to its previous value.
Py_CLEAR(tstate->frame) is wrong: it's a *borrowed* reference.
Pablo and me agree with PyThreadState_Clear() must not call
Py_CLEAR(tstate->frame): it's borrowed reference, not a strong reference.
I'm working on a PR to fix the issue.
----------
resolution: wont fix ->
status: closed -> open
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue20526>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
