I'm running a production Django application on two loadbalanced webservers and a single, dedicated Postgres server handling around 500k requests/day. I'm using memcached, and my database server performance has been fantastic.
Lately, I've been hitting very high percentages of free memory used, and occasionally spiking to very high percentages of swap memory used. I'm pretty familiar with Apache and Django, but not enough to diagnose what is happening - I'm hoping someone will be able to help fill in the gaps or tell me my understanding is off-base. Symptoms: What seems to be happening is that Apache grows to use all available memory, and my memused percentage will hit about 98%. From what I can tell, this does not necessarily mean that Apache is actively using 98% of memory. It is also my understanding that Apache won't automatically free up any memory until the process is killed. Perhaps someone could explain this? Is it strange that Apache never goes to 99%, instead always peaking at 98.x%? My swap % has spiked badly a few times, up to 69%. I'm told anything roughly > 5% is bad. Could inefficiently designed views lead to memory problems? I've made sure on my most popular pages that I'm only importing what is absolutely needed, and I've even dereferenced variables to try and help garbage collection. I grepped out the offending time period from my system activity report, and found the requests were low during the 10-20 minutes that the swap % spiked. I'm not sure that the request load was the culprit. Incidentals: My most popular view looks for a memcached RSS feed, and if it doesn't exist, uses URLLIB2 to go out and get it (then cache it). I've read about possible problems with URLLIB2.openurl() that it might not free the socket after use. I've made sure to close all connections and dereference all variables/objects before the view returns. Not sure if this is relevant, but it seems worth mentioning. My memcached hit ratio is 99%, so I'm comfortable that it's not opening a connection too often. What I've tried: I've tweaked one web server and left the other as-is for comparison. (both are running identical Django codebase) I've upgraded mod_python from 3.1 to 3.3.1, and I've upgraded Python from 2.4 to 2.5. Neither seems to have made a noticeable difference. I dropped down the max requests per child from 4k to 2.5k, and this shorter lifetime for the process seems to have helped. Questions: It is my understanding that Apache Prefork MPM is memory-intensive. Will my httpd processes keep growing in size until they are killed? Should I ever expect them to drop in size during their life? >From an application design perspective, are there any obvious culprits for high swap percentages in a Django project? Will increasing my memory from 1GB to 2GB help, or will Apache just swell to fill this new memory? ------------------------------------------------------- System: OS: Enterprise Red Hat 4 Memory: 1GB Apache: 2.0.2 Prefork MPM Python Version: 2.5 mod_python: 3.3.1 Django: .91-bug-fixes Caching: Memcached Relevant httpd.conf: Keepalives off StartServers 8 MinSpareServers 5 MaxSpareServers 15 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 2500 Recent Top Output: top - 12:36:23 up 108 days, 8:43, 1 user, load average: 0.01, 0.03, 0.00 Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 50.0% sy, 0.0% ni, 50.0% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1034636k total, 943036k used, 91600k free, 87320k buffers Swap: 1052216k total, 22560k used, 1029656k free, 177080k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5858 apache 16 0 72192 68m 404 S 0 6.8 0:05.70 memcached 5766 apache 15 0 70404 63m 3120 S 0 6.2 0:09.98 httpd 995 apache 15 0 69600 62m 3120 S 0 6.2 0:10.16 httpd 21319 apache 15 0 69424 62m 3112 S 0 6.2 0:06.73 httpd 25133 apache 15 0 69492 62m 3112 S 0 6.1 0:06.81 httpd 24232 apache 15 0 68784 61m 3112 S 0 6.1 0:06.42 httpd 25134 apache 16 0 68848 61m 3112 S 0 6.1 0:06.20 httpd 25132 apache 15 0 68368 61m 3112 S 0 6.0 0:06.24 httpd 29229 apache 16 0 59580 52m 3112 S 0 5.2 0:04.92 httpd 26287 apache 15 0 59232 52m 3112 S 0 5.2 0:04.95 httpd Sample of Bad Recent Sar -r output: 08:50:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 08:50:01 AM 143984 890652 86.08 10064 28176 962864 89352 8.49 22860 09:00:01 AM 15536 1019100 98.50 3608 25516 318740 733476 69.71 34576 09:10:01 AM 586352 448284 43.33 5384 39268 797504 254712 24.21 67148 09:20:01 AM 334088 700548 67.71 6596 39352 797520 254696 24.21 86132 09:30:01 AM 252024 782612 75.64 7600 39952 797832 254384 24.18 102208 09:40:01 AM 527368 507268 49.03 2976 24272 898520 153696 14.61 31512 09:50:01 AM 447144 587492 56.78 4184 24628 898520 153696 14.61 33848 10:00:01 AM 398904 635732 61.44 5244 25868 898520 153696 14.61 35188 10:10:01 AM 128408 906228 87.59 2952 31112 584540 467676 44.45 79816 10:20:01 AM 563540 471096 45.53 4344 33740 893044 159172 15.13 41996 10:30:01 AM 536080 498556 48.19 5332 33968 893044 159172 15.13 42600 10:40:01 AM 432272 602364 58.22 7316 41144 1022792 29424 2.80 11860 10:50:01 AM 576528 458108 44.28 8472 42068 1022792 29424 2.80 11860 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
