Graham Dumpleton wrote on 03/30/07 12:27: > On Mar 30, 6:31 pm, Steven Armstrong <[EMAIL PROTECTED]> wrote: >> My understanding is that every apache child pulls in all modules >> (mod_python, mod_perl, mod_php, mod_your_favorite_mod_here) which all >> consume memory. So you end up with loads of processes using ??MB of RAM >> to serve up e.g. a 1k image. >> >> I am by now means an expert on this. But maybe it's worth a shot. > > This doesn't really tell the whole story. Apache modules, like C > extensions for Python are implemented as shared objects, thus the code > segments of the Apache modules themselves should only incur a memory > hit once across all process. For example, the mod_php.so for some > versions of PHP is about 5MB. This does not mean though that each > Apache child process has 5MB of private memory dedicated just to it. > Instead their would generally only be a 5MB hit to the operating > system as a whole. Note though that I am only talking about code here > and not runtime memory usage. > > For mod_python.so though this unfortunately doesn't necessarily apply. > In the best case the mod_python.so file would be about 400KB in size > and it would be shared as described above. If your Python installation > was not built so as to generate a shared library though, when > mod_python.so is created the contents of the static Python library are > incorporated into the mod_python.so file itself, rather than being > linked at run time. As a result, the mod_python.so file can balloon > out to be 1.5-2.0MB in size, or much much more on certain operating > systems if debugger support was enabled when Python was compiled. > Worse is that because the object files from the Python static library > are not position independent, when the mod_python.so file is loaded > into memory, on some systems the operating system is forced to perform > address relocations on the code segments from the Python library > objects where they reside in mod_python.so. Because the address > relocations require modifying the memory it is no longer shared and as > a result it will actually end up consuming 1.5-2.0MB of private memory > in each Apache child process. > > Thus, one way of reducing memory usage with Apache is to ensure that > your Python is built with a shared library and not a static library > and that mod_python.so is actually using it. On Linux/Solaris systems, > running 'ldd' on mod_python.so will tell you. If you don't see a > dependency on libpythonX.Y.so then it isn't using Python as a shared > library and you will get this extra memory hit on every Apache child > process. > > This still isn't the end of the story. Because mod_python is a mix of > C code and Python code and it uses Python code as part of its dispatch > mechanism, when it is first loaded it loads a whole bunch of Python > modules before any request is even handled. It is because of these > Python modules being loaded that the startup memory usage of > mod_python appears to be so big. For a lot of these modules a web > application will generally load them in eventually anyway, so in the > bigger scheme of things it makes no difference, but for some modules a > web application may not use them so the memory usage is an unwelcome > hit. > > Worse is that recent investigations into mod_python show that some of > those Python modules are being loaded to get access to just a single > function. Other modules are being loaded all the time even though the > feature it is required for is optional and not often used. When > changes were made to mod_python to eliminate as many modules as one > could it actually dropped 1MB from the startup memory usage. > Realistically a web application might take back half of that later > when it wants the modules that were eliminated, but the modules for > the optional feature wouldn't. So, mod_python itself could see some > improvements in that area to potentially reduce memory usage as well. > > All in all though, this is still not going to help in the grand scheme > of things because Python web frameworks are just getting too fat. > Django is actually at the better end of the scale here as at minimum > it adds only about an extra 5MB of modules to size of each Apache > child process. TurboGears appears to be the worst, adding about 14MB, > with Pylons somewhere in the middle. > > So, although mod_python does have some things that can be cleaned up, > in the main it is the Python web frameworks themselves which add the > most to process size and that is before any code is run and data > caching becomes a factor. > > Graham >
Wow! Many thanks for the detailed explanation. Still, after reading your post, it seems one could save quite some resources by not serving static media trough the same apache instance which runs django and friends. I mean, usually you get a request, django does it's thing, sends back the page, which in turn requests another 10+ resources which don't require django or other server side processing. If those 10+ requests are handled by a lightweight server that could make quite a difference for a busy site. The section 'Scaling' at [1] has some other interesting info about all this. [1] http://www.djangobook.com/en/beta/chapter21/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
