[issue10370] py3 readlines() reports wrong offset for UnicodeDecodeError
New submission from Brian Warner : I noticed that the UnicodeDecodeError exception produced by trying to do open(fn).readlines() (i.e. using the default ASCII encoding) on a file that's actually UTF-8 reports the wrong offset for the first undecodeable character. From what I can tell, it reports (offset%4096) instead of the actual offset. I've attached a test case. It emits "all good" when run against py2.x (well, after converting the print() expressions back into statements), but reports an error at offset 4096 (reported as "0") on py3.1.2 and py3.2a3 . I'm running on a debian (sid) x86 box. The misreported offset does not occur with read(), just with readlines(). -- components: IO files: test.py messages: 120830 nosy: warner priority: normal severity: normal status: open title: py3 readlines() reports wrong offset for UnicodeDecodeError type: behavior versions: Python 3.1, Python 3.2 Added file: http://bugs.python.org/file19552/test.py ___ Python tracker <http://bugs.python.org/issue10370> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10370] py3 readlines() reports wrong offset for UnicodeDecodeError
Brian Warner added the comment: > Use .readline() to locate an invalid byte is not the right algorithm. If > you would like to do that, you should open the file in binary mode and > decodes the content yourself, chunk by chunk. Or if you manipulate small > files, you can use .read() as you wrote. Oh, I agree that readline() is inappropriate as a validation tool. My specific complaint is that the error message is misleading. I hit a message like this: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 26: ordinal not in range(128) and wanted to know if the file was UTF-8, or latin-1, or some other encoding, so I wanted to see that 0xe2 in context. The message said to look at offset 26, but the actual problem might be at 4122, or 8218, etc. It took me several minutes (and hexdump and grepping for ' e2 ') to find the character and figure out what was going on. Perhaps, if the error message cannot report a correct offset, then it shouldn't be reporting an offset at all. -- ___ Python tracker <http://bugs.python.org/issue10370> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
