[EMAIL PROTECTED] wrote on 03/29/07 20:15: > I'm running a production Django application on two loadbalanced > webservers and a single, dedicated Postgres server handling around > 500k requests/day. I'm using memcached, and my database server > performance has been fantastic. > > Lately, I've been hitting very high percentages of free memory used, > and occasionally spiking to very high percentages of swap memory used. > I'm pretty familiar with Apache and Django, but not enough to diagnose > what is happening - I'm hoping someone will be able to help fill in > the gaps or tell me my understanding is off-base. > > Symptoms: > > What seems to be happening is that Apache grows to use all available > memory, and my memused percentage will hit about 98%. From what I can > tell, this does not necessarily mean that Apache is actively using 98% > of memory. It is also my understanding that Apache won't automatically > free up any memory until the process is killed. Perhaps someone could > explain this? Is it strange that Apache never goes to 99%, instead > always peaking at 98.x%? > > My swap % has spiked badly a few times, up to 69%. I'm told anything > roughly > 5% is bad. Could inefficiently designed views lead to memory > problems? I've made sure on my most popular pages that I'm only > importing what is absolutely needed, and I've even dereferenced > variables to try and help garbage collection. > > I grepped out the offending time period from my system activity > report, and found the requests were low during the 10-20 minutes that > the swap % spiked. I'm not sure that the request load was the > culprit. > > Incidentals: > > My most popular view looks for a memcached RSS feed, and if it doesn't > exist, uses URLLIB2 to go out and get it (then cache it). I've read > about possible problems with URLLIB2.openurl() that it might not free > the socket after use. I've made sure to close all connections and > dereference all variables/objects before the view returns. Not sure if > this is relevant, but it seems worth mentioning. My memcached hit > ratio is 99%, so I'm comfortable that it's not opening a connection > too often. > > What I've tried: > > I've tweaked one web server and left the other as-is for comparison. > (both are running identical Django codebase) I've upgraded mod_python > from 3.1 to 3.3.1, and I've upgraded Python from 2.4 to 2.5. Neither > seems to have made a noticeable difference. I dropped down the max > requests per child from 4k to 2.5k, and this shorter lifetime for the > process seems to have helped. > > Questions: > > It is my understanding that Apache Prefork MPM is memory-intensive. > Will my httpd processes keep growing in size until they are killed? > Should I ever expect them to drop in size during their life? > >>From an application design perspective, are there any obvious culprits > for high swap percentages in a Django project? > > Will increasing my memory from 1GB to 2GB help, or will Apache just > swell to fill this new memory? > > ------------------------------------------------------- > > System: > > OS: Enterprise Red Hat 4 > Memory: 1GB > Apache: 2.0.2 Prefork MPM > Python Version: 2.5 > mod_python: 3.3.1 > Django: .91-bug-fixes > Caching: Memcached > > Relevant httpd.conf: > > Keepalives off > StartServers 8 > MinSpareServers 5 > MaxSpareServers 15 > ServerLimit 256 > MaxClients 256 > MaxRequestsPerChild 2500 > > Recent Top Output: > > top - 12:36:23 up 108 days, 8:43, 1 user, load average: 0.01, 0.03, > 0.00 > Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.0% us, 50.0% sy, 0.0% ni, 50.0% id, 0.0% wa, 0.0% hi, > 0.0% si > Mem: 1034636k total, 943036k used, 91600k free, 87320k > buffers > Swap: 1052216k total, 22560k used, 1029656k free, 177080k > cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 5858 apache 16 0 72192 68m 404 S 0 6.8 0:05.70 > memcached > 5766 apache 15 0 70404 63m 3120 S 0 6.2 0:09.98 > httpd > 995 apache 15 0 69600 62m 3120 S 0 6.2 0:10.16 > httpd > 21319 apache 15 0 69424 62m 3112 S 0 6.2 0:06.73 > httpd > 25133 apache 15 0 69492 62m 3112 S 0 6.1 0:06.81 > httpd > 24232 apache 15 0 68784 61m 3112 S 0 6.1 0:06.42 > httpd > 25134 apache 16 0 68848 61m 3112 S 0 6.1 0:06.20 > httpd > 25132 apache 15 0 68368 61m 3112 S 0 6.0 0:06.24 > httpd > 29229 apache 16 0 59580 52m 3112 S 0 5.2 0:04.92 > httpd > 26287 apache 15 0 59232 52m 3112 S 0 5.2 0:04.95 httpd > > Sample of Bad Recent Sar -r output: > > 08:50:01 AM kbmemfree kbmemused %memused kbbuffers kbcached > kbswpfree kbswpused %swpused kbswpcad > 08:50:01 AM 143984 890652 86.08 10064 28176 > 962864 89352 8.49 22860 > 09:00:01 AM 15536 1019100 98.50 3608 25516 > 318740 733476 69.71 34576 > 09:10:01 AM 586352 448284 43.33 5384 39268 > 797504 254712 24.21 67148 > 09:20:01 AM 334088 700548 67.71 6596 39352 > 797520 254696 24.21 86132 > 09:30:01 AM 252024 782612 75.64 7600 39952 > 797832 254384 24.18 102208 > 09:40:01 AM 527368 507268 49.03 2976 24272 > 898520 153696 14.61 31512 > 09:50:01 AM 447144 587492 56.78 4184 24628 > 898520 153696 14.61 33848 > 10:00:01 AM 398904 635732 61.44 5244 25868 > 898520 153696 14.61 35188 > 10:10:01 AM 128408 906228 87.59 2952 31112 > 584540 467676 44.45 79816 > 10:20:01 AM 563540 471096 45.53 4344 33740 > 893044 159172 15.13 41996 > 10:30:01 AM 536080 498556 48.19 5332 33968 > 893044 159172 15.13 42600 > 10:40:01 AM 432272 602364 58.22 7316 41144 > 1022792 29424 2.80 11860 > 10:50:01 AM 576528 458108 44.28 8472 42068 > 1022792 29424 2.80 11860 > >
Maybe it helps serving static media (images,css,) through e.g lighttpd instead of the same apache instance? My understanding is that every apache child pulls in all modules (mod_python, mod_perl, mod_php, mod_your_favorite_mod_here) which all consume memory. So you end up with loads of processes using ??MB of RAM to serve up e.g. a 1k image. I am by now means an expert on this. But maybe it's worth a shot. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
