On Jul 21, 1:03 pm, Michael Toomim <too...@gmail.com> wrote: > THANK YOU ALL SO MUCH for your help! > > I just learned a LOT. It looks like resource consumption was the > problem, because things are doing better on the bigger machine and > scaled down code. I've also added the MaxRequestsPerChild directive.
Are you using mod_wsgi embedded mode or daemon mode? That directive should not be required if you are using daemon mode of mod_wsgi. It is generally a bad idea to make arbitrary changes without understanding whether they are necessary. Changing to a larger machine without understanding why your application is using lots of memory in the first place is also questionable. All you have done is increased your head room but potentially not solved the original underlying problem. You may well just find that it all just blows up again when you get hit with a larger amount of traffic or a request which generates a lot of data. What are you going to do then, get an even bigger machine? Graham > I am soooooo happy to have this web server working, and very pleased > to know what to do when I hit a such a scaling wall again! > > And flask looks interesting, but I must say I really really like how > web2py's execfile puts things into global scope from the controllers > and automatically reloads code with each request. > > On Jul 20, 5:02 pm, Graham Dumpleton <graham.dumple...@gmail.com> > wrote: > > > > > On Jul 21, 8:18 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > Can you comment on memory usage? > > > > I have see this once: "after a while web serving slows" > > > it appeared to be due to a memory leak somewhere (did not experience > > > it with web2py+Rocket but only in web2py+mod_wsgi+apache). > > > > I googled it and I found Django was having the same problem on some > > > hosts: > > > Not sure how you can draw a parallel to that as it is a completely > > different framework and just because another framework, or more > > specifically one persons code, has issues, doesn't imply there is an > > issue with underlying web hosting. > > > These sorts of problems are isolated cases. If there was an issue with > > memory leakage in the hosting mechanism it would be affecting everyone > > and there have been no such reports of mod_wsgi itself leaking memory. > > That said, ensure you read: > > > http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html > > > This describes how Python itself leaks memory. For mod_wsgi 2.X and > > older, or if you are still loading mod_python into your Apache server, > > then you can be affected by this, but not if using mod_wsgi 3.X. > > > That post also explains how to completely disable initialisation of > > Python in Apache server child processes, ie., embedded, if you aren't > > using it. > > > >http://stackoverflow.com/questions/2293333/django-memory-usage-going-...... > > > > I followed the advice from a comment in the last post to limit the > > > number of requests served by each process: > > > Which is actually a useless thing to do if you are using daemon mode > > which I understood you were, as MaxRequestsPerChild directive only > > affects Apache server child process, ie., those use for embedded mode, > > and not daemon mode processes. > > > If using that directive helped and you were using daemon mode, then > > you likely have a memory leak in some other Apache module. > > > What you should have ensured you were doing was using display-name > > option to WSGIDaemonProcess to name the process. That way in 'ps' you > > can easily distinguish the mod_wsgi daemon mode processes from the > > Apache processes and work out which is leaking memory. If it is the > > daemon processes, it is likely to be a Python web application issue. > > If the Apache parent process is getting fatter and you perform a lot > > of Apache restart/reloads, then it could be that you are still using > > mod_wsgi 2.X or mod_python is loaded at same time, and you are using a > > version of Python that has lots of memory leaks on restarts. > > > If your daemon processes are not getting fat and the Apache server > > child processes are, then you may through incorrect configuration not > > even be running Python web application in daemon mode. This is where > > WSGIRestrictEmbedded as described in my post is good, as it will > > highlight when the configuration is screwed up. > > > > # prefork MPM > > > StartServers 5 > > > MinSpareServers 5 > > > MaxSpareServers 10 > > > MaxClients 256 > > > MaxRequestsPerChild 500 > > > ServerLimit 30 > > > > instead of the default: > > > > # prefork MPM > > > StartServers 5 > > > MinSpareServers 5 > > > MaxSpareServers 10 > > > MaxClients 256 > > > > The problem disappeared. The exact values that fix the problem may > > > depend on the ram available. > > > The other difference with above is that I think by setting ServerLimit > > to 30, you have effectively overridden MaxClients down to 30 even > > though set to 256. You have thus in part limited the exact problems > > described in: > > > http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa... > > > if it so happens you were using embedded mode and not daemon mode. > > > Graham > > > > Massimo > > > > On Jul 20, 4:30 pm, Michael Toomim <too...@gmail.com> wrote: > > > > > Let me also summarize the issues so far. > > > > > Originally: > > > > - I got three types of error messages in apache logs > > > > - Logging messages were often duplicated 2, 3, 5 times > > > > - I got the IOError ticket a few times > > > > - After a while the web serving slowed (some requests took up to a > > > > minute) and then quit completely > > > > > After rebooting: > > > > - I get one type of error message in apache logs, in big batches > > > > - I get the IOError ticket once or twice > > > > - After a while web serving slows (sometimes 150s per request) and > > > > stops > > > > > So I haven't been seeing the duplicate log messages anymore. > > > > > I upgraded to a bigger machine and am changing my code to remove ajax > > > > (will reduce load by 60x by decreasing functionality). I don't know > > > > what else to do. > > > > > On Jul 20, 2:03 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > Thanks for the clarification. > > > > > > @Michael, do you use the logging module? How? > > > > > > On Jul 20, 4:00 am, Graham Dumpleton <graham.dumple...@gmail.com> > > > > > wrote: > > > > > > > On Jul 20, 5:17 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > > > The problem with IOError, I can understand. As Graham says, if the > > > > > > > client closes the connection before the server responds or if the > > > > > > > server timesout the socket is closed and apache logs the IOError. > > > > > > > That isn't what I said. If you see that message when using daemon > > > > > > mode, the Apache server process that is proxying to the daemon > > > > > > process > > > > > > is crashing. This is different to the HTTP client closing the > > > > > > connection. You would only see that message if HTTP client closed > > > > > > connection if using embedded mode. > > > > > > > I know they are using daemon mode as that is the only situation > > > > > > where > > > > > > they could also see the message about premature end of script > > > > > > headers. > > > > > > > > What I really do not understand is why some requests are handled > > > > > > > by > > > > > > > multiple threads. web2py is agnostic to this (unless you use > > > > > > > Rocket > > > > > > > which you do not). web2py only provides a wsgi application which > > > > > > > is > > > > > > > executed - per thread - by the web server. It is the web server > > > > > > > (in > > > > > > > your case apache) that spans the thread, maps requests to threads, > > > > > > > calls the web2py wsgi application for each of them. > > > > > > > > If this is happening it is a problem with apache or with mod_wsgi. > > > > > > > More likely the problem is that they are registering the logging > > > > > > module from multiple places and that is why logging is displayed > > > > > > more > > > > > > than once. They should log the thread ID as well as that would > > > > > > confirm > > > > > > whether actually from the same thread where logging module handler > > > > > > has > > > > > > been registered multiple times. > > > > > > > Multiple registrations of logging handler could occur if it isn't > > > > > > done > > > > > > in a thread safe why, ie., so as to avoid multiple threads doing it > > > > > > at > > > > > > the same time. > > > > > > > Graham > > > > > > > > Can > > > > > > > you tell us more about the version of ubuntu, apache and mod_wsgi > > > > > > > that > > > > > > > you are using? Any additional information will be very useful. > > > > > > > > Massimo > > > > > > > > On Jul 19, 9:01 pm, Michael Toomim <too...@gmail.com> wrote: > > > > > > > > > I'm getting errors like these in my apache error logs: > > > > > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] > > > > > > > > Premature end > > > > > > > > of script headers: wsgihandler.py, > > > > > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > > > > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] > > > > > > > > Premature > > > > > > > > end of script headers: wsgihandler.py, > > > > > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > IOError: > > > > > > > > failed to write data > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > IOError: > > > > > > > > failed to write data > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > IOError: > > > > > > > > failed to write data > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI > > ... > > read more »
