On Mon, Sep 15, 2008 at 10:57 PM, JoeJ <[EMAIL PROTECTED]> wrote: > > 1- Data loaded in DB from: > http://leathergallery.com/module/data/sql/products/Products.sql -- > includes the phrase "Thinsulate(R)" > > 2- Django generic list_detail.object_list fails to render with: > Caught an exception while rendering: Could not decode to UTF-8 column > 'product_desc' with text 'Black gauntlet MC gloves with rain mittens > in a zip-closure pocket, reinforced palm and Thinsulate� lining.' >
This means the byte sequence stored in the database is not actually UTF-8. Somehow you've got non-UTF-8 encoded data stored in the database. How was this database built? > There has to be some encode/re.sub or something I can do on the way > into the DB ... or what ? > > This is my first real time using sqlite3. > It doesn't like valid SQL comments, it doesn't like backslashes to > escape chars, etc. -- annoying really. > > So, I'm guessing this is another sqlite3 annoyance. I'm an inch away > from just moving to mysql. > > I'm not too familiar with sqlite either, but have run across this problem. My (perhaps not entirely correct) understanding is that the database code doesn't actually care what encoding your data is in -- it accepts a sequence of bytes via a write call and will simply return that same sequence of bytes on a corresponding call to read the data. The problem arises when you have an interface layered on top of the database (such as pysqlite) that does try to actually interpret the data as being encoded in a particular way. Pysqlite tries to create a unicode object from the returned data assuming it is UTF-8 encoded. If that assumption is wrong, you get this error. Of course by the time you get this error, the damage has already been done, and you've got data you can't read in the DB. It would be better if the invalidly encoded data was rejected on the write. With the pysqlite included in Python 2.6, this is what happens (well, actually bare pysqlite will reject any non-ASCII bytestring, while Django using pysqlite still supports valid UTF-8 bytestrings). But prior to Python 2.6, it's apparently easy to supply non-UTF-8 encoded bytestrings and have them stored in the DB only to later find that they cannot be read. (They can be read by a lower-level interface that doesn't try to interpret the data as UTF-8, but I don't know enough about sqlite to give any instructions on how to do that.) So, probably not very helpful for fixing your problem but maybe uderstanding what is going on can help you in your decision on whether to stick with sqlite or use a different DB. Karen --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
