Hi all,
I'm using django in a server that is written as a twistd application. I've noticed that the memory footprint of the server grows heavily the longer it runs. When I commented out all sections calling django memory usage stays constant. So on first sight it seems the guilt lies with Django.
Below is the output of ps while running the server. As can be seen the percentage of memory used nearly triples in just a few seconds of cpu time. This is the result of calling the following function every ten seconds:
def check_jobs(self):
self.read_jobs()
open_jobs = Job.objects.all().filter(status__exact=2).order_by("due")
self.order_jobs(open_jobs)
reactor.callLater(10.0,
self.check_jobs)
where read_jobs() is a function that queries the database with a complex query (lots of Q's); it then goes on to call another function that does a lot of db stuff as well.
Most queries I make are either like
Job.objects.all().filter(status__exact=2) or loops over the elements of such a query set like
for job in jobs:
job.status = 3
job.save()
I intend to get rid of most of the latter kind by using the connection cursor directly.
The leak gets really bad over time, the server will end up eating as much memory as it can (we saw it gobbling down 1.5 GB) and eventually it will crash silently. I only have two global lists which I have checked with regard to keeping obsolete references. While I can't be sure, I don't think I'm keeping anything artificially alive. Can I tell django somehow to flush - I don't know - open connections, dis-cache objects or ...?
I don't know whether I've made much sense (it's been a long day and the deadline is looming and even faster approaching) but would appreciate any hint,
chris
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dwasrod 3487 48.7 0.7 18648 15164 ? Ss 22:48 0:01 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 21.6 0.7 18648 15164 ? Ss 22:48 0:01 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 17.2 0.7 19288 15704 ? Rs 22:48 0:02 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 20.7 0.8 20816 17180 ? Ss 22:48 0:03 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 16.4 0.8 20816 17180 ? Ss 22:48 0:03 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 20.3 0.9 22652 19064 ? Ss 22:48 0:05 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 17.3 0.9 22652 19064 ? Ss 22:48 0:05 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 16.3 0.9 22916 19544 ? Rs 22:48 0:06 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 17.9 1.0 24336 20812 ? Ss 22:48 0:07 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 15.7 1.0 24336 20812 ? Ss 22:48 0:07 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 17.9 1.0 26320 22684 ? Ss 22:48 0:09 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 16.4 1.0 26320 22684 ? Ss 22:48 0:09 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 15.6 1.1 26716 23200 ? Rs 22:48 0:10 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 16.9 1.1 28272 24704 ? Ss 22:48 0:11 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 15.7 1.1 28272 24704 ? Ss 22:48 0:11 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 17.2 1.2 30216 26792 ? Ss 22:48 0:13 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 16.2 1.2 30216 26796 ? Ss 22:48 0:13 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 15.7 1.3 30844 27288 ? Rs 22:48 0:14 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 16.6 1.3 32492 28848 ? Ss 22:48 0:15 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 15.8 1.3 32492 28848 ? Ss 22:48 0:15 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 16.9 1.4 34396 30816 ? Ss 22:48 0:17 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 16.1 1.4 34396 30816 ? Ss 22:48 0:17 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 15.6 1.5 35052 31344 ? Ss 22:48 0:17 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 16.4 1.5 36276 32864 ? Ss 22:48 0:19 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 15.8 1.5 36276 32864 ? Ss 22:48 0:19 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 16.7 1.6 38540 34988 ? Ss 22:48 0:21 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 16.1 1.6 38540 34988 ? Ss 22:48 0:21 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 15.6 1.7 39196 35452 ? Rs 22:48 0:21 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 16.3 1.7 40432 36980 ? Ss 22:48 0:23 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 15.8 1.7 40432 36980 ? Ss 22:48 0:23 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 16.5 1.8 42628 39020 ? Ss 22:48 0:25 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
dwasrod 3487 16.0 1.8 42628 39020 ? Ss 22:48 0:25 /usr/bin/python2.3 /usr/bin/twistd -o -y
jobs-application.py
dwasrod 3487 15.6 1.9 43024 39528 ? Ss 22:48 0:25 /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/django-users
-~----------~----~----~----~------~----~------~--~---
- Django memory leak? Christian Schneider
- Re: Django memory leak? Jeremy Dunck
- Re: Django memory leak? Christian Schneider
- Re: Django memory leak? Jeremy Dunck
- Re: Django memory leak? Malcolm Tredinnick
- Re: Django memory leak? Christian Schneider
