I have a Django app that does some heavy calculations for the user. At the start of a user's session it parses some source data and builds a complex data structure, in the form of big trees of cElementTree nodes (by themselves fast and small, being written in C). Then it allows the user to query and manipulate the data structure, using an AJAX interface.
I can't store the cElementTree data structure in Django's session, because it's made of unserializable C objects. I also guessed that de/ serializing a complex data structure at every AJAX request would seriously impact performance. So it needs to stay in RAM, in the form of Python and C objects. The way I'm doing it now, there is a global variable in one of my modules with a dict to map a user's session id with the data structure. A cleanup algorithm purges stale entries after a given timeout from the last access. This forced me to deploy the project with method=threaded, to ensure there is only one of those cache dicts. I've given some thought to Python's GIL and the kind of bottleneck it must be in a method=threaded deployment (in a high-load scenario, with many users waiting for an answer and many CPU cores to be used, otherwise it would make no difference). The only reasonable alternative I see is to use pure Python ElementTree objects (instead of the unserializable C ones) and store them in Django's sessions over memcached. This would allow for a method=prefork deployment. Whether the lack of inter-user Python locking would offset the cost of using a somewhat larger and slower data structure and un/pickling it at every request, remains to be seen. Any thoughts? Tobia -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
