Warning: almost no actual Django-related content below... On Thu, 2007-03-29 at 18:15 +0000, [EMAIL PROTECTED] wrote: > I'm running a production Django application on two loadbalanced > webservers and a single, dedicated Postgres server handling around > 500k requests/day. I'm using memcached, and my database server > performance has been fantastic. > > Lately, I've been hitting very high percentages of free memory used, > and occasionally spiking to very high percentages of swap memory used. > I'm pretty familiar with Apache and Django, but not enough to diagnose > what is happening - I'm hoping someone will be able to help fill in > the gaps or tell me my understanding is off-base. > > Symptoms: > > What seems to be happening is that Apache grows to use all available > memory, and my memused percentage will hit about 98%. From what I can > tell, this does not necessarily mean that Apache is actively using 98% > of memory. It is also my understanding that Apache won't automatically > free up any memory until the process is killed.
That's fairly normal: the free() call in C will often keep the memory for the process's further use on a subsequent malloc() call. So memory usage appears to only go up and never down. > Perhaps someone could > explain this? Is it strange that Apache never goes to 99%, instead > always peaking at 98.x%? I'm not familiar enough with Apache internals to know about this. However, at some point, being able to effectively utilise 99.x% of memory is difficult because of fragmentation of the memory blocks. A lot of malloc()/free() pairs for differing sizes could lead to fragmentation over time. I don't know for sure that this is what's happening, but it's possible and so what you're seeing is not unbelievable. > My swap % has spiked badly a few times, up to 69%. I'm told anything > roughly > 5% is bad. An actual percentage of swap usage isn't really indicative of anything much. The real performance problem is when swap is actually being swapped in and out. Because that is very slow compared to memory accesses (it has to do a disk access). So if your machine is actively swapping, the performance is going to go down. If it has just moved some very rarely accessed stuff into swap, that isn't necessarily a performance hit because that stuff could be just sitting there unaccessed, so it's never being waited on. On a server, though, if swap has non-trivial amounts of stuff in it, you have to wonder how it got there. On a desktop it might happen because you have multiple apps open and are only using a few at once. On a server that is less likely -- all applications are generally in active use. So keep an eye on swap usage, but particularly pay attention to active swapping, which is the real performance killer and a sign of something using a lot of memory. > Could inefficiently designed views lead to memory > problems? I've made sure on my most popular pages that I'm only > importing what is absolutely needed, and I've even dereferenced > variables to try and help garbage collection. Unlikely. Profiling actively running Python processes for memory usage is a bear, though. So it's going to be very hard to tell. You'll end up having to poke around in places like /proc/<fd>/maps (on a Linux system). A tool like pcat from The Coroner's Toolkit is very useful here, too. Be aware that tracking down what is really using the memory takes time and is often a futile task if there is no single large user. [...] > I've tweaked one web server and left the other as-is for comparison. > (both are running identical Django codebase) I've upgraded mod_python > from 3.1 to 3.3.1, and I've upgraded Python from 2.4 to 2.5. Neither > seems to have made a noticeable difference. I dropped down the max > requests per child from 4k to 2.5k, and this shorter lifetime for the > process seems to have helped. This is a lot higher than the default value. I've seen some pretty high volume (e.g. internet banking) sites running happily with the default of 1000 for MaxRequestsPerChild. Do you have a reason for cranking this up so high, or it was just an experiment? Quite possibly there's logic in your settings -- you clearly seem to have done your research -- but I would be trying to keep this number down a bit more. Memory management is one of the main reasons MaxRequestsPerChild exists -- it mitigates any creeping memory leaks, for example -- so if you are seeing excess usage over time, restarting the child processes more frequently is one solution. Juggling this number and MinSpareServers requires some experimenting and knowing the usage patterns of your server (# simultaneous requests, burstiness behaviour, etc). Based on the sar output you showed, it looks like the number of servers times the amount of memory they want is too much for your system, so I would definitely be looking to reduce one of those numbers. Restarting the processes more frequently is a way to reduce the second number. > Questions: > > It is my understanding that Apache Prefork MPM is memory-intensive. > Will my httpd processes keep growing in size until they are killed? > Should I ever expect them to drop in size during their life? > > >From an application design perspective, are there any obvious culprits > for high swap percentages in a Django project? Not that I can think of. There's very little persistent information between requests (the imported Python code is basically nothing). Memchached is obviously going to use some memory. > > Will increasing my memory from 1GB to 2GB help, or will Apache just > swell to fill this new memory? Realise that having all your memory used (or nearly used) is not a bad thing. Many server processes are designed to allow them to grow to fill available space (and not require more, so they can operate with more as well as less memory). It just occurred to me that you haven't mentioned whether you are actually seeing performance problems or not. If you aren't seeing performance problems, then I'm not sure there is a real problem here -- Apache is probably behaving fairly normally. So, unfortunately, this isn't really helping diagnose the root cause. I'm just suggesting some mitigating measures. If you really want to delve deeply, break out the Coroner's Toolkit and go crazy. It's fun, it's educational, it's often frustrating (oops :( ), but it's sometimes necessary. Regards, Malcolm --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
