"Michael Lewis" <mikelikes...@gmail.com> writes: > I'm using Python to sanitize my logs from invalid UTF8 characters before > COPYing them into postgres. I came across this one sequence that seems to > be valid UTF8 (in the extended range I believe).
It is not valid. See http://tools.ietf.org/html/rfc3629 --- a sequence beginning with ED must have a second byte in the range 80-9F to be legal, and this doesn't. The example you give would decode as U+DF2D, ie part of a surrogate pair, which is specifically disallowed in UTF8 --- you're supposed to code the original character directly, not via a surrogate pair. The primary reason for this rule is that otherwise there are multiple ways to encode the same character, which can be a security hazard. > It goes through both pythons encoding as well as iconv without an error You should file bugs against those tools. regards, tom lane -- Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-bugs
