New submission from Nick Coghlan: For PEP 538, setting PYTHONIOENCODING turned out to have undesirable side effects on Python 2 instances in subprocesses, since Python 2 has no 'surrogateescape' error handler.
So I switched to using the "Py_SetStandardStreamEncoding" API defined in http://bugs.python.org/issue16129 instead, but this turns out to have problematic interactions with the dynamic memory allocator management, so it fails with a fatal exception in debug mode. An example of the error can be seen here: https://travis-ci.org/python/cpython/jobs/211293576 The problem appears to be that between the allocation of the memory with `_PyMem_RawStrdup` in `Py_SetStandardStreamEncoding` and the release of that memory in `initstdio`, the active memory manager has changed (at least in a debug build), so the deallocation as part of the interpreter startup fails. That interpretation is based on this comment in Programs/python.c: ``` /* Force again malloc() allocator to release memory blocks allocated before Py_Main() */ (void)_PyMem_SetupAllocators("malloc"); ``` The allocations in Py_SetStandardStreamEncoding happen before the call to Py_Main/Py_Initialize, but the deallocation happens in Py_Initialize. The "fix" I applied to the PEP branch was to make the default allocator conditional in Programs/python.c as well: ``` #ifdef Py_DEBUG (void)_PyMem_SetupAllocators("malloc_debug"); # else (void)_PyMem_SetupAllocators("malloc"); # endif ``` While that works (at least in the absence of a PYTHONMALLOC setting) it seems fragile. It would be nicer if there was a way for Py_SetStandardStreamEncoding to indicate which allocator should be used for the deallocation. ---------- messages: 289668 nosy: haypo, ncoghlan priority: normal severity: normal stage: needs patch status: open title: Py_SetStandardStreamEncoding leads to a memory error in debug mode type: crash _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29818> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
