Hi all
I'm encountering a difficult to solve unicode problem whilst saving data to
the database. Worst of all, any attempt to reduce it to a simple test case,
or reproduce it in the console fail(!). This is on django 1.0.
The process encountering the error is a simple daemon, run from a management
command [1]. The process looks up a task [2] to run and executes it. After
the task has finished executing, it updates the generated_content member on
the model, either to contain any pertinent error messages if there was a
failure, or to store rendered HTML if the task was successful.
The problem occurs when the generated HTML contains particular unicode
characters (in this case, right single quotation mark, \u2019), which for
some reason prompts django or MySQLdb to decide to convert it to unicode.
The unicode HTML comes from rendering a django template; here's the snippet
that generates the HTML:
cdict = { ... } # left out; template renders correctly, so not
important..
ctxt = Context(cdict)
from django.template import loader
content = loader.render_to_string('the_template.html',
context_instance=ctxt)
self.task.generated_content = content
This code is called from MigrationTask::execute() - this is in the (working)
PerformMigration class - and is the last thing that happens before we call
save() on the modified instance. Apart from the generated_content, the only
other thing that changes on this model as a result of this code is the
status attribute.
When we do call save(), the following traceback is produced:
Traceback (most recent call last):
File
"/usr/local/www/django/ssosp/externals/identity_provider/tasks/management/commands/taskrunner.py",
line 44, in handle
task.execute()
File
"/usr/local/www/django/ssosp/externals/identity_provider/tasks/models.py",
line 39, in execute
self.save()
File
"/usr/local/www/django/ssosp/root/lib/python2.5/site-packages/django/db/models/base.py",
line 307, in save
self.save_base(force_insert=force_insert, force_update=force_update)
File
"/usr/local/www/django/ssosp/root/lib/python2.5/site-packages/django/db/models/base.py",
line 358, in save_base
rows = manager.filter(pk=pk_val)._update(values)
File
"/usr/local/www/django/ssosp/root/lib/python2.5/site-packages/django/db/models/query.py",
line 429, in _update
return query.execute_sql(None)
File
"/usr/local/www/django/ssosp/root/lib/python2.5/site-packages/django/db/models/sql/subqueries.py",
line 117, in execute_sql
cursor = super(UpdateQuery, self).execute_sql(result_type)
File
"/usr/local/www/django/ssosp/root/lib/python2.5/site-packages/django/db/models/sql/query.py",
line 1700, in execute_sql
cursor.execute(sql, params)
File
"/usr/local/www/django/ssosp/root/lib/python2.5/site-packages/django/db/backends/mysql/base.py",
line 83, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.5/site-packages/MySQLdb/cursors.py", line
151, in execute
query = query % db.literal(args)
File "/usr/local/lib/python2.5/site-packages/MySQLdb/connections.py", line
247, in literal
return self.escape(o, self.encoders)
File "/usr/local/lib/python2.5/site-packages/MySQLdb/connections.py", line
180, in string_literal
return db.string_literal(obj)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 1182: ordinal not in range(128)
If I set a break point where we generate the content, print out
repr(content), copy paste that into a django python shell and assign it to a
task's generated_content property, it saves correctly.
If I manually change content to u'\u2019' inside the debugger, it also saves
correctly. It also works correctly for u'\u2019'*2048, just in case size of
string matters.
The database and all tables are set to UTF-8 in mysql. My locale is
correctly set up in both cases (en_GB.UTF-8). I'm very confused as to why it
is attempting to convert it to ascii :/
Any hints/tips greatly appreciated.
Cheers
Tom
[1] http://pastebin.com/m9e23563
[2] http://pastebin.com/m564e1cd7
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=.
