Re: Questions about UnicodeDecodeError

2012-04-03 Thread Ian Clelland
On Monday, April 2, 2012, Ali Mesdaq wrote: > I did have UTF-8 in the db before but then when I was inserting(done by a > script) values in I would get errors. I really want to avoid doing > conversions before I insert the data so I can maintain the data in the > exact format as it was seen in.

Re: Questions about UnicodeDecodeError

2012-04-02 Thread Bill Freeman
Keeping things as seen is the point. If the data contains any characters outside of the ASCII range, then you can't (directly) store it (reliably) in a db set for ASCII. Unicode encompasses all the characters that you will find on the web. UTF-8 is a relatively trouble free way to store unicode

Re: Questions about UnicodeDecodeError

2012-04-02 Thread Ali Mesdaq
I did have UTF-8 in the db before but then when I was inserting(done by a script) values in I would get errors. I really want to avoid doing conversions before I insert the data so I can maintain the data in the exact format as it was seen in. I may be forced into doing that it seems to make django

Re: Questions about UnicodeDecodeError

2012-04-01 Thread Bill Freeman
If you can, switch to UTF-8 in the db. Web traffic can arrive in a variety of encodings, and while the headers tell the server how to make unicode out of it, but by having the db set for ASCII, you limit what you can store (since not all unicode characters can be converted to ASCII. On Fri, Mar 3

Questions about UnicodeDecodeError

2012-03-30 Thread Ali Mesdaq
I have a situation where I am reading data that I have no control over and inserting it into a db. The data is http headers. I am storing them in postgres db in a text field and the db encoding is SQL_ASCII. Since the data can be anything even non compliant http headers with anything for its values