On Thu, May 28, 2009 at 4:57 AM, Stefan Tunsch <stun...@gmail.com> wrote:
>
> Hi!
>
> I have a rather strange set up, for running a django intranet site.
>
> It is currently running on a Windows XP box with Apache 2.2 with
> mod_python and MySQL.
>
> The site is rather prone with database queries. The most complex page on
> the site reports to be executing around 350 sql queries.
> The site is done with everything in one single django application.
> The views of the site are spread across two files taking close under
> 2000 lines of code.
>
> I do not cache in any way.
>
> This is so, because the site is more of a management application than a
> normal public site. There are many parts of each page that are
> calculations that have to be exact and have to be calculated on each
> request.
>
> In this scenario, I've ended up having a site capable of getting the CPU
> of the box down to it's knees on EACH request.
Not too surprising with 2000 LOC and 350 SQL queries.
>
> Each single request will get the CPU to 100% and the page won't
> completely render before 5-7 seconds have passed. When more persons
> interact with the server, it isn't capable of responding appropriately
> anymore.
>
> The curious thing is that it's not MySQL that is responsible of all that
> CPU usage, but the Apache process.
> The same thing happens if I load the site with the development server
> via manage.py runserver.
>
I am going to explain this with an example. Imagine I have a database that
has 300,000 widgets in it. If I were to do Widget.objects.all() and load
them all into python, that means I have 300,000 python objects loaded into
memory. The SQL statement would be insanely simple and quickly executed but
the python would need a ton of memory to hold all of those objects.
>
> It's also not my views which are causing the trouble.
> Setting up a timer at the start and end of the most complicated view
> shows that the app gets through that part in approximately 0.2-0.3 seconds.
>
> It seems clearly that the rendering of the templates is responsible for
> eating up all of the CPU's resources.
If you are cutting off the timer before render_to_response, you are probably
missing most (if not all) of the SQL queries. This is because django queries
are lazy and not executed until they absolutely have to be. A lot of times
this means they are executed when you call attributes in the templates.
>
> One thing I have done since the start of the project, and which might
> be part of the problem, has been to end all the views like this:
> return render_to_response('myapp/mytemplate.html', locals(),
> context_instance=RequestContext(request))
>
> I'm passing the locals() variable dict to the template and also the
> request.
>
> This post is to ask for some guidance or ideas on how to tackle the
> problem.
> I hope there is a simple and obvious reason for this performance
> bottleneck...
>
> I have started to cache, (currently via the in-memory cache option, to
> keep it simple). The approach is to use the "cache" template tag to
> cache portions of my templates which don't need to be refreshed on each
> request and which do require calls to the database.
>
Caching will help but if one request will bring the system to its knees, you
will still have problems.
350 queries is quite a lot, and I bet you are bringing a lot of data onto
the page. I wouldn't be surprised to hear that the page is extremely large
as well (>200kb just in the HTML). This would also not be very efficient
for Apache or the browser. If it is that large, I would look into turning
the page into multiple pages, or look into pagination, which will help you
out with objects in memory as well.
Caching certain parts of the page might also help you out. You can cache
fragments of the template. You could also create globals that will stay in
the python process as long as it lives, but that might increase memory usage
as well.
You can also put a debugger in place and see if there is anything specific
that might be causing the slow down.
I hope that helps a little,
Michael
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
