Im sorry about that. The reason I did not post same day was that through the messages in this thread someone suggested adding a debug statement to debug WebappClassLoader, and I wanted to wait to see if it happened again so I could see the debug output, then I came up with the idea in my last message. yes we are on Linux and we are using java 7 and tomcat 7 and we are on linux 3.11.6 on a 64 bit build for x86. When I say crash, I mean that when we use the client application to connect to the web app it wont connect and we have to go into tomcat manager and start it back up. We use an executor with 20 threads in it to process things on the server, and that was never shut down, so that would explain the memory leak and the extra threads. We need them to run for the duration of the application life cycle, but they should be cleaned up on undeployment. Thanks for the reference to the presentation. I will go through it now. I think I may have my answer and hopefully this wont happen again after we fix our web app to shut down those threads.
On Fri, Apr 4, 2014 at 11:17 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Elias, > > On 4/4/14, 11:03 AM, Elias Kopsiaftis wrote: > > Ive done some more research into this problem with my co developer, > > and we have a question. Is it possible that memory leaks would > > cause tomcat to crash? > > Yes, but you'll likely get an OutOfMemoryError in your log, and then > everything will go crazy. Some requests will work, others will not. > Things will get slow. For us, the background thread often quits and > sessions never expire, exacerbating the problem. > > But it won't happen silently (you may have to look for it) and it > won't spontaneously undeploy and redeploy your web application. > > > Silly question I know, but heres technically what the situation > > is. > > > > We did not add code to shut down all threads when application is > > undeployed through tomcat manager. > > Tomcat will do this itself: you don't need to add any code. Or did you > mean that you have Threads created by your application that continue > to tun after your web app has been shut down? > > > When undeploying and redeploying we cause memory leaks due to our > > application. > > Yes, this can happen. The Threads cause the WebappClassLoader to stay > in memory, which means that all Class objects for the previous > deployment are still hanging around. It can be several tens of MiB per > reload. > > You should read this excellent presentation which explains the > problem(s) and the solution(s): > > http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf > > > Now, I guess the real question is, we seem to always deploy it > > fine, but randomly at some point during the night it will crash, > > and we only know that from checking next morning. > > What do you mean when you say "crash"? Exception? Spontaneous > re-deployment? OutOfMemory? JVM crash (process exits)? > > > We have nothing logged that indicates why. > > If the process is gone without any indication of why (no shutdown > message, no nothing), my only guess is Linux OOM-killer. You never > actually came back to describe the environment you are in after saying > you'd post that information "later today." If you're not on Linux.... > I'm not sure. > > > Would memory leaks cause it to crash randomly even though nothing > > is trying to connect to it, yet it would seem to deploy fine? > > I've never seen an OOME actually cause a JVM to quit. It just slogs > onward, often not able to accomplish any useful work, but the process > stays running. > > > Could that be the issue we are seeing? Is there any specific class > > to debug in logging.properties that might indicate whats going on? > > I added that line you guys mentioned about setting log level to > > DEBUG from that class loader class, but it hasnt crashed since > > then.... > > Describe what you mean by "crash", including as much details about it, > and definitely read the presentation referenced above. Also definitely > fix your web application to shut down its Threads. Whatever component > launches them (e.g. ServletContextListener, Servlet.init, etc.) should > stop them in the appropriate callback method > (ServletContextListener.contextDestroyed, Servlet.destroy, etc.). > > - -chris > > > On Thu, Apr 3, 2014 at 12:06 PM, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Mark, > > > > On 4/2/14, 5:20 PM, Mark Eggers wrote: > >>>> Chris, > >>>> > >>>> On 4/2/2014 1:54 PM, Christopher Schultz wrote: > >>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 > >>>>> > >>>>> Mark, > >>>>> > >>>>> On 4/2/14, 4:30 PM, Mark Eggers wrote: > >>>>>> Chris, > >>>>>> > >>>>>> On 4/2/2014 1:05 PM, Christopher Schultz wrote: > >>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 > >>>>>>> > >>>>>>> Chuck, > >>>>>>> > >>>>>>> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: > >>>>>>>>> From: Elias Kopsiaftis [mailto:yemi...@gmail.com] > >>>>>>>>> Subject: tomcat randomly undeploys and redeploys > >>>>>>>>> the applications > >>>>>>>> > >>>>>>>>> I deploy the application, then in the log file > >>>>>>>>> catalina.out i get many messages from > >>>>>>>>> WebappClassLoader clearReferencesThreads saying > >>>>>>>>> threads appear to have started but have failed to > >>>>>>>>> stop > >>>>>>>> > >>>>>>>> This is an indication that your webapp is not > >>>>>>>> managing its threads properly. > >>>>>>>> > >>>>>>>>> then finally, Ill get a message from HostConfig > >>>>>>>>> checkResources that says its undeploying the > >>>>>>>>> context, and then it redeploys. > >>>>>>>> > >>>>>>>> This is sometimes caused by incorrect timestamps on > >>>>>>>> the bits of the webapp that Tomcat monitors, or an > >>>>>>>> incorrect clock setting on the system Tomcat is > >>>>>>>> running on. The mismatch makes it appear that the > >>>>>>>> webapp is being updated continuously. > >>>>>>> > >>>>>>> I've found that in development, a single update can > >>>>>>> cause Tomcat to go into a loop of redeployments, > >>>>>>> re-deploying my web application every few seconds or > >>>>>>> so. If I let it go, it does finally stop reloading and > >>>>>>> settle down. > >>>>>>> > >>>>>> > >>>>>> Can you describe your development environment a little > >>>>>> bit, and any thoughts as to what might trigger this loop > >>>>>> of redeployments? > >>>>> > >>>>> I use Eclipse for development, but our "real" build process > >>>>> is ant-based. We have some watched resources configured > >>>>> outside the default (stuff like Struts config files, > >>>>> etc.). > >>>>> > >>>> > >>>> Ah, makes sense. > >>>> > >>>>> When I do a build while Tomcat is running, usually I get > >>>>> one webapp reload, but sometimes I get a series of reloads. > >>>>> It usually gets so irritating (our webapp takes about 10 > >>>>> seconds to reload) that I just kill Tomcat and immediately > >>>>> restart it. It starts up once and all is well after that. > >>>>> > >>>> > >>>> Yep, and in the process more files are copied about, and > >>>> that triggers another reload. > >>>> > >>>> Fun, fun. > > > > No, the deployment update takes like one or two seconds. It's > > usually something like copying less than 10 class files or > > whatever. It's nearly instantaneous. Whatever happens, it's not > > because I'm updating files during the reload. I could understand > > that situation. > > > > What I observe is that I update my application, I wait maybe 10 > > seconds, and then Tomcat reloads my application multiple times > > before I just kill it. > > > >>>>>> I've not seen this, but it could explain some issues some > >>>>>> the developers I support are seeing. > >>>>> > >>>>> It definitely happens, and I never remember to enable the > >>>>> DEBUG logging to find out what resource it thinks has been > >>>>> updated until after it happens, at which point I just don't > >>>>> care. Perhaps I should enable it right now :) > >>>>> > >>>>> - -chris > >>>> > >>>> I've managed to make this happen in my environment now > >>>> (NetBeans 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be > >>>> upgraded soon). I just needed an application that takes a bit > >>>> longer to load. I only managed to trigger two reloads, so > >>>> it's not much of an issue. > >>>> > >>>> Maybe look at adding the backgroundProcessorDelay attribute > >>>> to the context? I don't know what would happen if the context > >>>> got a string of reload requests within the delay interval. > >>>> Would it queue them up one after the other, or would it just > >>>> execute one? > > > > I think it's more important to see what file(s) Tomcat thinks > > is(are) being updated. I wonder if it's the same file, or if > > there's some weird timestamp issue happening. Perhaps there is even > > some kind of edge case where a resource's last-modified date isn't > > being updated properly. > > > > In most cases, Tomcat reloads my application a single time, as > > expected. These reload-storms are fairly rare. > > > > -chris > >> > >> --------------------------------------------------------------------- > >> > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJTPs0fAAoJEBzwKT+lPKRYYBcQAI+xdEUBXVdylZIWxcJoY097 > v9xgkd6Zud4QoupVhUVrdP7YCjjhFTtS0y9MCOMxLdSyQQFjKS+yFfjE8Xtyah7x > tnUwf200qnUBVCaBgDY+lfmrx+AuSNogce81qqpQkYUGiVJT+rW0MYnMQfgeCPmg > RchOe6zo5664bvU3EBRnoPe3UYZcIwVjK9vXS8dy9vMygZJ6zGN+XIs0PTlFE5h/ > HA8Ik6UJoK6yvyHqOfY2wC0xXeKMUsbNAcsIBcuC4H7h/NbDQCoH0YV5vvkpOBhF > vhMLvCtzOgTEK0oLDBsbuiAXF6g1XzQk+Uxoy0a1Kx5U4TbswFIK2hHtAtmQWC1y > v8RvvvNYNRO3tzI3Zo/CFO3+4Ha5EHmj96HLJI2itIOefD+By+y+gqqUR35lGdNT > 7NY4zd0fIYQqkqkBajpDPJX6eGShEHX0obF5dR/46uaehl+Vy9tfUq+fZEILP2Cm > XZdi6SD0epzpX+J6/7usQErU4QIxAnmmxwHu+ShhvRMF0+5ZsWCkt5RvZxMNuiMQ > g9aDC1T+RACVOTCe80WDKgOnWmeGhbCycClb2ITSt3ceh/tb+JOShJoIgNl8r1MN > Hq1h/aNxYLJRsKt3c4P+PXui4bgh8JVe3gtAuilo1s+0Z7gBgaAmwIVtG3gDA/hH > ERlGxSO1fiY+0/p3Lf+f > =3VV0 > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
