New submission from Ma Lin: While I was trying to optimize GB2312/GBK/GB18030-2000 codecs (they are three encodings that widely used in China), I found there is a bug.
The three encodings, their relation should be: GB2312 ⊂ GBK ⊂ GB18030-2000. However, in Python's implement: GB2312 ⊄ GBK ⊂ GB18030-2000. GBK should be backward compatible with GB2312, but in Python's implement, it's not. ---- I digged into, I found the Python's GB2312 codec is using a wrong convert table. In this file /Modules/cjkcodecs/_codecs_cn.c , there is a comment block, I paste it here: /* GBK and GB2312 map differently in few code points that are listed below: * * gb2312 gbk * A1A4 U+30FB KATAKANA MIDDLE DOT U+00B7 MIDDLE DOT * A1AA U+2015 HORIZONTAL BAR U+2014 EM DASH * A844 undefined U+2015 HORIZONTAL BAR */ In fact the second column (GB2312 column) is wrong, this column should be deleted. The four involved unicode codepoints are: U+30FB ・ KATAKANA MIDDLE DOT U+00B7 · MIDDLE DOT U+2015 ― HORIZONTAL BAR U+2014 — EM DASH So, GB2312 codec decodes b'0xA1, 0xA4' to U+30FB. U+30FB is a Japanese symbol, but looks quite similar to U+00B7. I searched "GB2312 Unicode Table" with Google, there are right verson and wrong version on the Internet, unfortunately we are using the wrong verson. libiconv-1.14 is also using the wrong version. ---- Hold an example of bad behavior. Using GBK encoder, encode U+30FB to bytes, UnicodeEncodeError exception occurred, becase U+30FB is not in GBK. In Simplified Chinese version of Microsoft Windows, console's default encoding is GBK[1]. If using GB2312 decoder to decode b'0xA1, 0xA4', then print U+30FB to console, UnicodeEncodeError raised. Since DASH is a common character, this bug is annoying. ---- If we fix this, I don't know how many disasters will happen. However, if we don't fix this, it's a bug. I already made a patch, but I think we need a discussion, should we fix this? ----------------------- Annotate: [1] In fact console's default encoding is cp936, cp936 almost same as GBK, but not entirely same. Using GBK in here is not a problem. ---------- components: Unicode files: fixgb2312.patch keywords: patch messages: 241858 nosy: Ma Lin, ezio.melotti, haypo priority: normal severity: normal status: open title: GB2312 codec is using a wrong covert table type: behavior versions: Python 3.5, Python 3.6 Added file: http://bugs.python.org/file39182/fixgb2312.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24036> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
