On Wed, Jun 3, 2009 at 1:22 AM, proteus...@gmail.com <proteus...@gmail.com> wrote: > > We've got an application that is running behind apache, mod_wsgi and > front ending a postgres 8.3 db. We needed to upgrade our Satchmo to > the latest version which requires Django 1.1. After the upgrade the > system starts eventually running out of memory until it thrashes and > locks up the machine under a fairly light load, just a few reqs per > second. I can understand the system using more memory but it doesn't > seem to release it when the load stops. > > This graph: http://www.flickr.com/photos/proteusguy/3589788760 shows > the results of both systems which, besides the Django & Satchmo > versions are identical. You can see that when the test completes on > the old system the process returns a great amount of memory but the > new one never releases the memory. In production this will eventually > grow to consume all RAM. This particular test is of a set of 7 > different requests simulating what a typical user might do. > > This graph: http://www.flickr.com/photos/proteusguy/3589788752 shows a > similar test except it only contains a single request out of the seven > from the prior test. We were hoping to identify that something in one > of our requests was the cause of the problem but similar exclusive > test runs with all 7 requests from the combined test from above show > similar results - fairly stable memory usage. > > This leads me to believe that the issue is systemic to django, most > likely in the db handling as most of the requests don't touch the > Satchmo system. If this was a C++ app I'd be able to figure this out > pretty quick but I'm not sure how to attack the problem in a dynamic > language environment like python. So this leaves me with two > questions: > > 1. Is anyone else seeing this kind of memory extra memory usage and an > inability to return memory once requests end?
I'm not seeing anything like this on my deployed sites, running a fairly recent (post-beta-1) checkout of Django. > 2. What suggestions does anyone have about how I can determine what > aspect of the system is using this extra memory and holding onto it? The best advice I can point you at is Ned Batchelder's blog, when he diagnosed a memory leak he found when moving from v0.96 to v1.0: http://nedbatchelder.com/blog/200809/a_server_memory_leak.html As well as describing a general approach, Ned mentions a few tools, and the comments mention a few more. However, I would say that my experience has been that Python code is very rarely the cause of memory leaks such as these. You have to try pretty hard to leak in a garbage collected language. You can do it, but the causes are generally obvious - usually things like code that has lots of intensive looping causing circular references in data structures, or global data structures that aren't cleaned up. In Django, for example, a common source of a "memory leak" is the global query log that is maintained when DEBUG=True on the server. However, this isn't really a "leak" in the classical sense - the server is just keeping hold of everything it was told to store. If you tell Django not to store every query, then it wont, and you won't "leak" memory. A more likely cause is any C extension module that you are interfacing with. Any Django stack includes at least one C module - the database adaptor. Even if you haven't changed the database adaptor, this is still an area worth investigating - it's possible that in the transition from Django 1.0 to 1.1, the database adaptor is being exercised in a slightly different way, and as a result is exposing a leak that wasn't previously a problem. If you can find the source of the memory leak, and it does turn out to be a problem with something that Django has introduced since 1.0, fixing it will be a high priority for us. Unfortunately, I can't think of anything we have done that is an immediate or obvious candidate. While there have been lots of changes to the DB handling for v1.1, I can't think of anything that would be a candidate for leaking memory. Any assistance you can provide (even if it's just at the level of isolating a test case that leaks reliably) will be gratefully accepted. Yours, Russ Magee %-) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
