Ah, preventing multithreading is a good idea to try too. It wasn't a file descriptor problem either, I had Files used: 1376 out of 75556
On Jul 20, 9:14 pm, Graham Dumpleton <graham.dumple...@gmail.com> wrote: > On Jul 21, 1:41 pm, Michael Toomim <too...@gmail.com> wrote: > > > I'm using daemon mode... I didn't realize that the directive won't > > matter in daemon mode. > > > Yes, I think I probably will run into the problem again when I get > > more usage. However, I'm still not convinced it's a memory problem, > > because I had 30mb free on my 740mb machine when I was having the > > problem, with 0 swap usage. > > Well, as I explained before, it perhaps is a resource leakage such as > file descriptors. You can exhaust kernel file descriptors and still > have lots of memory available. > > I have seen various cases before for different peoples applications on > different frameworks where file objects weren't being explicitly > closed, or database connection pools not being managed properly, such > that the number of open file descriptors went up and up and eventually > they ran out. This can cause all sorts of weird errors to manifest > when it occurs, including it impacting other applications if is > exhausted system wide. For example, in a shell, not being able to > execute commands to even debug the problem. > > I would suggest you become familiar with some of the basic monitoring > commands such as 'lsof' or 'ofiles', depending on what your system > provides. You can then use these to monitor file descriptor usage by > your processes. > > Also be aware that such problems may only arise when multithreading > kicks in and concurrent requests run. In other words, due to code > which isn't thread safe. If you don't get the concurrency, you may > well see your application run quite happily. > > Thus, one suggestion is to not use multiple threads for daemon mode > processes and instead use something like 'processes=5 threads=1'. This > will avoid the potential of it being caused by multithreading issues > at least. > > Graham > > > > > I don't know what I'll do if I this happens again. My code just does > > simple database lookups and updates, it doesn't create circular > > references nor store anything in global variables, so if there's a > > memory leak I worry it's somewhere further up the stack. I don't know > > any ways to investigate memory consumption to see where it's being > > used. > > > On Jul 20, 8:23 pm, Graham Dumpleton <graham.dumple...@gmail.com> > > wrote: > > > > On Jul 21, 1:03 pm, Michael Toomim <too...@gmail.com> wrote: > > > > > THANK YOU ALL SO MUCH for your help! > > > > > I just learned a LOT. It looks like resource consumption was the > > > > problem, because things are doing better on the bigger machine and > > > > scaled down code. I've also added the MaxRequestsPerChild directive. > > > > Are you using mod_wsgi embedded mode or daemon mode? That directive > > > should not be required if you are using daemon mode of mod_wsgi. > > > > It is generally a bad idea to make arbitrary changes without > > > understanding whether they are necessary. Changing to a larger machine > > > without understanding why your application is using lots of memory in > > > the first place is also questionable. All you have done is increased > > > your head room but potentially not solved the original underlying > > > problem. You may well just find that it all just blows up again when > > > you get hit with a larger amount of traffic or a request which > > > generates a lot of data. What are you going to do then, get an even > > > bigger machine? > > > > Graham > > > > > I am soooooo happy to have this web server working, and very pleased > > > > to know what to do when I hit a such a scaling wall again! > > > > > And flask looks interesting, but I must say I really really like how > > > > web2py's execfile puts things into global scope from the controllers > > > > and automatically reloads code with each request. > > > > > On Jul 20, 5:02 pm, Graham Dumpleton <graham.dumple...@gmail.com> > > > > wrote: > > > > > > On Jul 21, 8:18 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > > Can you comment on memory usage? > > > > > > > I have see this once: "after a while web serving slows" > > > > > > it appeared to be due to a memory leak somewhere (did not experience > > > > > > it with web2py+Rocket but only in web2py+mod_wsgi+apache). > > > > > > > I googled it and I found Django was having the same problem on some > > > > > > hosts: > > > > > > Not sure how you can draw a parallel to that as it is a completely > > > > > different framework and just because another framework, or more > > > > > specifically one persons code, has issues, doesn't imply there is an > > > > > issue with underlying web hosting. > > > > > > These sorts of problems are isolated cases. If there was an issue with > > > > > memory leakage in the hosting mechanism it would be affecting everyone > > > > > and there have been no such reports of mod_wsgi itself leaking memory. > > > > > That said, ensure you read: > > > > > > http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html > > > > > > This describes how Python itself leaks memory. For mod_wsgi 2.X and > > > > > older, or if you are still loading mod_python into your Apache server, > > > > > then you can be affected by this, but not if using mod_wsgi 3.X. > > > > > > That post also explains how to completely disable initialisation of > > > > > Python in Apache server child processes, ie., embedded, if you aren't > > > > > using it. > > > > > > >http://stackoverflow.com/questions/2293333/django-memory-usage-going-...... > > > > > > > I followed the advice from a comment in the last post to limit the > > > > > > number of requests served by each process: > > > > > > Which is actually a useless thing to do if you are using daemon mode > > > > > which I understood you were, as MaxRequestsPerChild directive only > > > > > affects Apache server child process, ie., those use for embedded mode, > > > > > and not daemon mode processes. > > > > > > If using that directive helped and you were using daemon mode, then > > > > > you likely have a memory leak in some other Apache module. > > > > > > What you should have ensured you were doing was using display-name > > > > > option to WSGIDaemonProcess to name the process. That way in 'ps' you > > > > > can easily distinguish the mod_wsgi daemon mode processes from the > > > > > Apache processes and work out which is leaking memory. If it is the > > > > > daemon processes, it is likely to be a Python web application issue. > > > > > If the Apache parent process is getting fatter and you perform a lot > > > > > of Apache restart/reloads, then it could be that you are still using > > > > > mod_wsgi 2.X or mod_python is loaded at same time, and you are using a > > > > > version of Python that has lots of memory leaks on restarts. > > > > > > If your daemon processes are not getting fat and the Apache server > > > > > child processes are, then you may through incorrect configuration not > > > > > even be running Python web application in daemon mode. This is where > > > > > WSGIRestrictEmbedded as described in my post is good, as it will > > > > > highlight when the configuration is screwed up. > > > > > > > # prefork MPM > > > > > > StartServers 5 > > > > > > MinSpareServers 5 > > > > > > MaxSpareServers 10 > > > > > > MaxClients 256 > > > > > > MaxRequestsPerChild 500 > > > > > > ServerLimit 30 > > > > > > > instead of the default: > > > > > > > # prefork MPM > > > > > > StartServers 5 > > > > > > MinSpareServers 5 > > > > > > MaxSpareServers 10 > > > > > > MaxClients 256 > > > > > > > The problem disappeared. The exact values that fix the problem may > > > > > > depend on the ram available. > > > > > > The other difference with above is that I think by setting ServerLimit > > > > > to 30, you have effectively overridden MaxClients down to 30 even > > > > > though set to 256. You have thus in part limited the exact problems > > > > > described in: > > > > > > http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa... > > > > > > if it so happens you were using embedded mode and not daemon mode. > > > > > > Graham > > > > > > > Massimo > > > > > > > On Jul 20, 4:30 pm, Michael Toomim <too...@gmail.com> wrote: > > > > > > > > Let me also summarize the issues so far. > > > > > > > > Originally: > > > > > > > - I got three types of error messages in apache logs > > > > > > > - Logging messages were often duplicated 2, 3, 5 times > > > > > > > - I got the IOError ticket a few times > > > > > > > - After a while the web serving slowed (some requests took up > > > > > > > to a > > > > > > > minute) and then quit completely > > > > > > > > After rebooting: > > > > > > > - I get one type of error message in apache logs, in big batches > > > > > > > - I get the IOError ticket once or twice > > > > > > > - After a while web serving slows (sometimes 150s per request) > > > > > > > and > > > > > > > stops > > > > > > > > So I haven't been seeing the duplicate log messages anymore. > > > > > > > > I upgraded to a bigger machine and am changing my code to remove > > > > > > > ajax > > > > > > > (will reduce load by 60x by decreasing functionality). I don't > > > > > > > know > > > > > > > what else to do. > > > > > > > > On Jul 20, 2:03 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > > > > Thanks for the clarification. > > > > > > > > > @Michael, do you use the logging module? How? > > > > > > > > > On Jul 20, 4:00 am, Graham Dumpleton > > > > > > > > <graham.dumple...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > On Jul 20, 5:17 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > > > > > > The problem with IOError, I can understand. As Graham says, > > > > > > > > > > if the > > > > > > > > > > client closes the connection before the server responds or > > > > > > > > > > if the > > > > > > > > > > server timesout the socket is closed and apache logs the > > > > > > > > > > IOError. > > > > > > > > > > That isn't what I said. If you see that message when using > > > > > > > > > daemon > > > > > > > > > mode, the Apache server process that is proxying to the > > > > > > > > > daemon process > > > > > > > > > is crashing. This is different to the HTTP client closing the > > > > > > > > > connection. You would only see that message if HTTP... > > read more »
