Just in case someone else stumbles across this old thread. TLDR: The recipe may be able to be improved - I only needed to close the connection in the parent Python process, not in the child process.
-- Best guess hypothesis as to what's going on is when you fork a process, the OS (e.g. OSX Mountain Lion for me) will simply copy the memory, including any current DB connection details. Then the child process in Django sometimes tries to reuse it. I say sometimes because Django's backend.mysql.base.py tries: self.connection.ping() If that fails, it closes the connection, then the Django magic of creating a new connection to the default DB will happen later when it is needed. Reusing a connection then sometimes leads to one or more whacky things in the child processes: - DatabaseError: (2013, 'Lost connection to MySQL server during query') - DatabaseError: (2006, 'MySQL server has gone away') - MultipleObjectsReturned: get() returned more than one <Model object -- it returned 6! Lookup parameters were ... - DoesNotExist: <Model object> matching query does not exist. Perhaps someone will be motivated one day to patch something like this into Django itself, but it feels like something that comes with the territory of handling multiprocessing without using a meaningful subset of concurrency such as RabbitMQ/Celery for a task queue. In other words, there's a reason Python's multiprocessing documentation is filled with many notes and warnings - concurrency is one of the more difficult to use constructs in the developer toolbox and as developers we should use it only when it is justified by specific requirements for extreme performance (or perhaps for the personal challenge). On Saturday, March 7, 2009 2:30:47 AM UTC+11, Jirka Vejrazka wrote: > > > I have a management command that that starts up a few process to process > a > > request: http://dpaste.com/7925/. > > Hi, I was recently debugging similar issue and came to a conclusion > (which may be wrong of course :) that multiprocessing and Django DB > connections don't play well together. I ended up closing Django DB > connection first thing in the new process. It'll recreate a new > connection when it needs one, but that one will have no references to > the connection used by the parent. > > So, my Process.start() calls a function which starts with: > >>> from django.db import connection > >>> connection.close() > > This solved my problem. > > Cheers > > Jirka > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/-u-g-YzdJTQJ. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
