[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Walter Dörwald added the comment: The original specification (PEP 293) required that an error handler called for encoding *must* return a replacement string (not bytes). This returned string must then be encoded again. Only if this fails an exception must be raised. Returning bytes from the e

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Marc-Andre Lemburg added the comment: On 29.09.2021 10:41, Serhiy Storchaka wrote: > > Restricting the returned position to be strictly larger than start would > solve the problem with infinite loop and OOM. But this is a different issue. Yes, this would make sense, since having the codec pr

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Serhiy Storchaka added the comment: Restricting the returned position to be strictly larger than start would solve the problem with infinite loop and OOM. But this is a different issue. -- ___ Python tracker __

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Change by Marc-Andre Lemburg : -- nosy: +doerwalter ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Change by Marc-Andre Lemburg : -- components: +Unicode nosy: +ezio.melotti ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Marc-Andre Lemburg added the comment: Looking at the specs in PEP 293 (https://www.python.org/dev/peps/pep-0293/), it is certainly possible for the error handler to return a newpos outside the range start - end, meaning in most cases: a value >= end. There's a good reason for this: the codec

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

STINNER Victor added the comment: > We could just forbid error handlers returning position not in the range > (start , end] Yeah, that sounds like a reasonable solution. I don't see the point of returning a position outside this range. What would be the use case? For me, the only corner cas

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Change by Serhiy Storchaka : -- pull_requests: +26973 pull_request: https://github.com/python/cpython/pull/28593 ___ Python tracker ___

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Serhiy Storchaka added the comment: I am working on it, since it is more complex issue, and PR 13134 does not solve it. 1. This bug affects also other codecs implemented in C: ASCII, Latin1, UTF-8, etc. 2. It still crashes in UTF-16/32 encoders if the error handler returns a position less t

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Irit Katriel added the comment: Reproduced on 3.11. -- nosy: +iritkatriel versions: +Python 3.10, Python 3.11, Python 3.9 -Python 3.7, Python 3.8 ___ Python tracker ___ __

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

anthony shaw added the comment: Easily reproduced on master, thanks (lldb) run encode_crash.py Process 14743 launched: '/Users/anthonyshaw/repo/cpython/python.exe' (x86_64) Objects/unicodeobject.c:448: _PyUnicode_CheckConsistency: Assertion "((PyObject*)(op))->ob_type))->tp_flags & ((1UL

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Change by Andrei Talaba : Added file: https://bugs.python.org/file48305/encode_crash.py ___ Python tracker ___ ___ Python-bugs-list mailing

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Change by Andrei Talaba : Removed file: https://bugs.python.org/file48304/encode_crash.py ___ Python tracker ___ ___ Python-bugs-list mailin

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Change by Serhiy Storchaka : -- assignee: -> serhiy.storchaka versions: -Python 3.6 ___ Python tracker ___ ___ Python-bugs-list ma

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

Change by Andrei Talaba : -- keywords: +patch pull_requests: +13047 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-l

[issue36819] Crash during encoding using UTF-16/32 and custom error handler

New submission from Andrei Talaba : The CPython interpreter write out-of-bounds of allocated memory in certain edge cases in the utf-16 and utf-32 encoders. The attached script registers two error handlers that either write one ascii character, or two bytes, and tells the encoder to start ag