[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Changes by Raymond Hettinger : -- resolution: -> fixed status: open -> closed type: -> behavior versions: +Python 2.7, Python 3.5 ___ Python tracker ___ ___

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Roundup Robot added the comment: New changeset fc6d62db8d42 by Raymond Hettinger in branch '2.7': Issue #25135: Avoid possible reentrancy issues in deque_clear. https://hg.python.org/cpython/rev/fc6d62db8d42 -- ___ Python tracker

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Roundup Robot added the comment: New changeset 005590a4a005 by Raymond Hettinger in branch '3.5': Issue #25135: Avoid possible reentrancy issues in deque_clear. https://hg.python.org/cpython/rev/005590a4a005 -- nosy: +python-dev ___ Python tracker

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Raymond Hettinger added the comment: Thanks Timmy! -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Tim Peters added the comment: The only way to be certain you're never going to face re-entrancy issues in the future is to call malloc() directly - and hope nobody redefines that too with some goofy macro ;-) In the meantime, stick to PyMem_Malloc(). That's the intended way for code holding

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Raymond Hettinger added the comment: > I'm not sure that you understand what you are trying to do. The goal is to avoid possible re-entrancy both now and in the future. Since PyMem_RawMalloc is thread-safe and doesn't require the GIL to be held, it is guaranteed that there won't be a callback

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Serhiy Storchaka added the comment: Here is a patch that copies only used part of the block. I don't see an actual bug in patch 2 besides that it fallbacks to current popping behavior. -- Added file: http://bugs.python.org/file40572/deque_nonreentrant_clear4.diff _

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

STINNER Victor added the comment: "Victor, I was thinking of switching to PyMem_RawMalloc instead of PyMem_Malloc. The advantages are that there are guaranteed to be no side-effects, the GIL doesn't need to be held, and there is less overhead. Are there any disadvantage I should know about?"

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Raymond Hettinger added the comment: Victor, I was thinking of switching to PyMem_RawMalloc instead of PyMem_Malloc. The advantages are that there are guaranteed to be no side-effects, the GIL doesn't need to be held, and there is less overhead. Are there any disadvantage I should know about

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Serhiy Storchaka added the comment: I perhaps were overcautious. There is a difference between the case of deque and the case of lru_cache and OrderedDict. The latter creates Python object (PyDict_New) that can trigger garbage collecting, and the former calls only PyMem_Malloc. The latter can

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Raymond Hettinger added the comment: Serhiy contends that calling newblock() can mutate the deque (its only external call is PyMem_Malloc()). I'm trying understand how that could be possible if it is just a thin wrapper around malloc. Before gumming-up the code in an effort to avoid calling n

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Serhiy Storchaka added the comment: Here is a patch that doesn't need newblock(). -- Added file: http://bugs.python.org/file40494/deque_nonreentrant_clear3.diff ___ Python tracker __

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

STINNER Victor added the comment: "Does PyMem_Malloc() do anything other than call malloc()? We hold the GIL so there doesn't seem to be a need to use PyMem_RawMalloc()." Well, the difference between PyMem_Malloc() and PyMem_RawMalloc() is subtle. Right now, it should only impact debug tools

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Raymond Hettinger added the comment: How does newblock() mutate the deque? Does PyMem_Malloc() do anything other than call malloc()? We hold the GIL so there doesn't seem to be a need to use PyMem_RawMalloc(). -- ___ Python tracker

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Serhiy Storchaka added the comment: I left comments on Rietveld. Perhaps you missed Rietveld messages in the Spam folder. -- nosy: +serhiy.storchaka ___ Python tracker ___ _

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Changes by Raymond Hettinger : Added file: http://bugs.python.org/file40486/deque_nonreentrant_clear2.diff ___ Python tracker ___ ___ Python-b

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

Changes by Raymond Hettinger : -- keywords: +patch Added file: http://bugs.python.org/file40477/deque_nonreentrant_clear.diff ___ Python tracker ___ _

[issue25135] Deques to adopt the standard clearing procedure for mutable objects

New submission from Raymond Hettinger: The clear method for deques is possibly reentrant. Use the safer technique used in listobject.c, setobject.c, and dictobject.c. -- assignee: rhettinger components: Extension Modules messages: 250811 nosy: rhettinger priority: normal severity: norm