On Wed, 2006-05-24 at 23:22 +0200, Christian Schneider wrote:
> Hi all,
>
> I'm using django in a server that is written as a twistd application.
> I've noticed that the memory footprint of the server grows heavily the
> longer it runs. When I commented out all sections calling django
> memory usage stays constant. So on first sight it seems the guilt lies
> with Django.
>
> Below is the output of ps while running the server. As can be seen the
> percentage of memory used nearly triples in just a few seconds of cpu
> time. This is the result of calling the following function every ten
> seconds:
>
> def check_jobs(self):
> self.read_jobs()
> open_jobs =
> Job.objects.all().filter(status__exact=2).order_by("due")
> self.order_jobs(open_jobs)
> reactor.callLater(10.0, self.check_jobs)
>
> where read_jobs() is a function that queries the database with a
> complex query (lots of Q's); it then goes on to call another function
> that does a lot of db stuff as well.
>
> Most queries I make are either like
> Job.objects.all().filter(status__exact=2) or loops over the elements
> of such a query set like
>
> for job in jobs:
> job.status = 3
> job.save()
>
> I intend to get rid of most of the latter kind by using the connection
> cursor directly.
>
> The leak gets really bad over time, the server will end up eating as
> much memory as it can (we saw it gobbling down 1.5 GB) and eventually
> it will crash silently. I only have two global lists which I have
> checked with regard to keeping obsolete references. While I can't be
> sure, I don't think I'm keeping anything artificially alive. Can I
> tell django somehow to flush - I don't know - open connections,
> dis-cache objects or ...?
You're going to see a lot more memory usage in these situations if you
have DEBUG = True. Because Django is saving a copy of every SQL query
(the query itself, not the results) and it looks like you are doing a
truckload of them. This one isn't a leak, it's a debugging aid.
If you want to see if this is the full cause of your problems and you
still need DEBUG = True, you could (a) hack django/db/backends/util.py
and remove the appending to db.queries in CursorDebugWrapper.execute(),
or (b) call django.db.reset_queries() periodically, which flushes the
connection.queries list.
If you have DEBUG = False, we need to look elsewhere
Regards,
Malcolm
>
> I don't know whether I've made much sense (it's been a long day and
> the deadline is looming and even faster approaching) but would
> appreciate any hint,
>
> chris
>
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> dwasrod 3487 48.7 0.7 18648 15164 ? Ss 22:48
> 0:01 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 21.6 0.7 18648 15164 ? Ss 22:48
> 0:01 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 17.2 0.7 19288 15704 ? Rs 22:48
> 0:02 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 20.7 0.8 20816 17180 ? Ss 22:48
> 0:03 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.4 0.8 20816 17180 ? Ss 22:48
> 0:03 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 20.3 0.9 22652 19064 ? Ss 22:48
> 0:05 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 17.3 0.9 22652 19064 ? Ss 22:48
> 0:05 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.3 0.9 22916 19544 ? Rs 22:48
> 0:06 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 17.9 1.0 24336 20812 ? Ss 22:48
> 0:07 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.7 1.0 24336 20812 ? Ss 22:48
> 0:07 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 17.9 1.0 26320 22684 ? Ss 22:48
> 0:09 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.4 1.0 26320 22684 ? Ss 22:48
> 0:09 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.6 1.1 26716 23200 ? Rs 22:48
> 0:10 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.9 1.1 28272 24704 ? Ss 22:48
> 0:11 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.7 1.1 28272 24704 ? Ss 22:48
> 0:11 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 17.2 1.2 30216 26792 ? Ss 22:48
> 0:13 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.2 1.2 30216 26796 ? Ss 22:48
> 0:13 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.7 1.3 30844 27288 ? Rs 22:48
> 0:14 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.6 1.3 32492 28848 ? Ss 22:48
> 0:15 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.8 1.3 32492 28848 ? Ss 22:48
> 0:15 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.9 1.4 34396 30816 ? Ss 22:48
> 0:17 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.1 1.4 34396 30816 ? Ss 22:48
> 0:17 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.6 1.5 35052 31344 ? Ss 22:48
> 0:17 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.4 1.5 36276 32864 ? Ss 22:48
> 0:19 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.8 1.5 36276 32864 ? Ss 22:48
> 0:19 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.7 1.6 38540 34988 ? Ss 22:48
> 0:21 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.1 1.6 38540 34988 ? Ss 22:48
> 0:21 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.6 1.7 39196 35452 ? Rs 22:48
> 0:21 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.3 1.7 40432 36980 ? Ss 22:48
> 0:23 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.8 1.7 40432 36980 ? Ss 22:48
> 0:23 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.5 1.8 42628 39020 ? Ss 22:48
> 0:25 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 16.0 1.8 42628 39020 ? Ss 22:48
> 0:25 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
> dwasrod 3487 15.6 1.9 43024 39528 ? Ss 22:48
> 0:25 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
>
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users
-~----------~----~----~----~------~----~------~--~---
