[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Ned Deily added the comment: New changeset 30c2ae4dcfd19acbdfb7045151c73d5700eec7b4 by Ned Deily (Miss Islington (bot)) in branch '3.7': [3.7] bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304) (GH-14369) https://github.com/python/cpython/commit/30c2ae4dcfd19acbdfb7045151

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Ned Deily added the comment: @xtreak, If you set the priority to "release blocker", the appropriate release managers will be nosied automatically. I'd rather see too many potential "release blocker"s than too few. -- ___ Python tracker

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Karthikeyan Singaravelan added the comment: Thanks Ned for the details, in future reports I will also try to add the respective release manager to the issue in case of regressions. -- ___ Python tracker ___

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Ned Deily added the comment: "The only changes allowed to occur in a maintenance branch without debate are bug fixes. Also, a general rule for maintenance branches is that compatibility must not be broken at any point between sibling minor releases (3.5.1, 3.5.2, etc.). For both rules, only

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Karthikeyan Singaravelan added the comment: Thanks Ned, is there a general policy on regressions to be marked as release blocker, like if a regression that was made in 3.7.3 then it acts as a release blocker for 3.7.4 or is it based on severity? ? --

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Ned Deily added the comment: Since the latest fix (GH-14369) addresses a regression between 3.7.3 and 3.7.4rc1, I am going to cherry-pick it into 3.7.4 final. (In retrospect, this regression should have been marked as a "release blocker".) -- nosy: +ned.deily __

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

STINNER Victor added the comment: Thanks Karthikeyan Singaravelan for the bug report of the regression. You're right that I misunderstood it. Thanks Serhiy for the second fix. The regression should now be fixed as well, I close the issue again. -- resolution: -> fixed stage: patch r

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

STINNER Victor added the comment: New changeset c755ca89c75252a7aae9beae82fd47787a76b9e2 by Victor Stinner (Miss Islington (bot)) in branch '3.7': [3.7] bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304) (GH-14369) https://github.com/python/cpython/commit/c755ca89c75252a7

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

miss-islington added the comment: New changeset d32594ad27f48a898d42a0ea30b9d007b1c57de9 by Miss Islington (bot) in branch '3.8': bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304) https://github.com/python/cpython/commit/d32594ad27f48a898d42a0ea30b9d007b1c57de9

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by miss-islington : -- pull_requests: +14186 pull_request: https://github.com/python/cpython/pull/14369 ___ Python tracker ___ __

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by miss-islington : -- pull_requests: +14185 pull_request: https://github.com/python/cpython/pull/14368 ___ Python tracker ___ __

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Serhiy Storchaka added the comment: New changeset 894263ba80af4b7733c2df95b527e96953922656 by Serhiy Storchaka in branch 'master': bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304) https://github.com/python/cpython/commit/894263ba80af4b7733c2df95b527e96953922656 ---

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Serhiy Storchaka added the comment: Victor, I think you misunderstood the issue. The problem is not that a decoding error is raised. The problem is that the incremental decoder no longer raises where it raised before. I think that both behavior may be correct, and that it is better to not re

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Serhiy Storchaka : -- pull_requests: +14128 pull_request: https://github.com/python/cpython/pull/14304 ___ Python tracker ___

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

STINNER Victor added the comment: > UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 1: > invalid continuation byte Python is right: b'f\xf1\xf6rd' is not a valid UTF-8 string: $ python3 Python 3.7.3 (default, May 11 2019, 00:38:04) >>> b'f\xf1\xf6rd'.decode() Traceback

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Roufique Hossain : -- nosy: +roufique7 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by STINNER Victor : -- resolution: fixed -> status: closed -> open ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Karthikeyan Singaravelan added the comment: This change seems to have caused test failure reported in https://github.com/python-hyper/wsproto/issues/126 from codecs import getincrementaldecoder decoder = getincrementaldecoder("utf-8")() print(decoder.decode(b'f\xf1\xf6rd', False)) # With thi

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Serhiy Storchaka added the comment: New changeset bd48280cb66544827952ca91e326cbb178c8c461 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7': bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603) (GH-12627) https://github.com/python/cpython/commit/bd48280cb66544827952ca91e326cbb

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by miss-islington : -- pull_requests: +12560 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Serhiy Storchaka added the comment: New changeset 7a465cb5ee7e298cae626ace1fc3e7d97df79f2e by Serhiy Storchaka in branch 'master': bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603) https://github.com/python/cpython/commit/7a465cb5ee7e298cae626ace1fc3e7d97df79f2e -- ___

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Serhiy Storchaka added the comment: PR 12603 fixes this issue in more general way and does not affect performance. -- ___ Python tracker ___ __

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Serhiy Storchaka : -- versions: +Python 3.7 -Python 3.9 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Serhiy Storchaka : -- pull_requests: +12543 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Serhiy Storchaka : -- versions: +Python 3.8, Python 3.9 -Python 3.5, Python 3.6 ___ Python tracker ___ ___ Python-bugs-lis

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Serhiy Storchaka : -- assignee: -> serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Change by Inada Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Serhiy Storchaka added the comment: The patch slows down decoding up to 20%. $ ./python -m timeit -s 'b = b"\xc4\x80"*1' -- 'b.decode()' Unpatched: 1 loops, best of 3: 50.8 usec per loop Patched:1 loops, best of 3: 63.3 usec per loop And I'm not sure that fixing only for the su

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka stage: -> patch review versions: +Python 3.5 -Python 3.4 ___ Python tracker ___

[issue24214] UTF-8 incremental decoder doesn't support surrogatepass correctly

Changes by STINNER Victor : -- title: Exception with utf-8, surrogatepass and incremental decoding -> UTF-8 incremental decoder doesn't support surrogatepass correctly ___ Python tracker __