Hi Chris, Thanks for the help. I don't see any OOME around this time. Before restarting I collected the heap and thread dumps. Looking at one of the thread dumps, I see the ContainerBackgroundProcessor (below). So looks like that thread didn't die entirely, may be it just stopped working ?
*"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007fd000a5d800 nid=0x9be5 waiting on condition [0x00007fcfc4955000]* * java.lang.Thread.State: TIMED_WAITING (sleeping)* * at java.lang.Thread.sleep(Native Method)* * at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1508)* * at java.lang.Thread.run(Thread.java:745)* * Locked ownable synchronizers:* * - None* - I have attached a screenshot of the StandardManager runtime attributes from the heapdump. Is there anything anyone can see wrong here? - Is there anything else I can look for in the heapdump and threaddump? [image: Inline image 1] thanks, Stephen On Fri, Jun 3, 2016 at 5:57 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Stephen, > > On 6/3/16 8:47 PM, Stephen Bhadran wrote: > > *Linux version:* 2.6.32-279.5.2.el6.x86_64 ( > > mockbu...@c6b10.bsys.dev.centos.org) (gcc version 4.4.6 20120305 > > (Red Hat 4.4.6-4) (GCC) ) *Java Version: *1.7.0_55 *Tomcat > > Version: *7.0.40 *Session Timeout: *30 minutes > > > > Hi, Two of our servers reached max heap and we had to restart to > > resolve the issue. > > > > In troubleshooting, we noticed Http Sessions had stopped expiring > > 05/12th and reached 25K over a period of 11 days. I noticed the > > same behavior in both servers, the session expiration seem to have > > stopped on both of them around the same time. I'm saying this > > because from looking at the monitoring graphs (NewRelic) I notice > > the http sessions starting to climb on both servers around the same > > time frame. > > > > I am thinking, the issue could be two folds: #1 - The session > > expiration stopped working -OR #2 - The sessions themselves were > > created with TTLs way into the future > > > > I suspected #1. I looked that tomcat 7.0.40 source code, extracted > > all session/manager error messages and searched for them in the > > logs and didn't find any hits. I was hoping to find something about > > "Session expiration stopped working", but didn't find any. > > > > Can you please advise how to troubleshoot this issue? I'd very > > much appreciate the help. > > Were there any errors around the 12th? I've found that in rare cases, > the server hits an OOME while the BackgroundProcessor thread is > running, which kills the BackgroundProcessor. > > Guess which thread is responsible for triggering the session-cleaner? > > If the BackgroundProcessor died, it should have dumped an error to a > log file. As I said, it's pretty rare, but it does happen and then > it's only a matter of time before your sessions bust your heap. > > You can manually-trigger the session-cleaner via JMX of you absolutely > have to, but I'd personally prefer to bounce a Tomcat instance in that > case. > > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJXUid5AAoJEBzwKT+lPKRYJfYQAKA2ditjB7B2vLbm0po83Tr0 > YD2EjqLBrF4O/3Wu6enIDRAQEIGgXmTr2bgPlMvu+6aoDopiCSKltRxWQFlTMyIP > 8ura/0YlS5sX3yGPPKN5nrZnOz2dvrgPncHLNVfuM9BEm+8QZPeKNZVDhMH53m1v > 4YcLCM5IjswAS4FrVeG9A0gcZplBd53WUuypnRYQMUcS1sACdItoZ3TLAwb/WBg/ > T9yoyftEXJVtP6lovkaWka3yw3v382P6HmSV6/TtCWjogB/tBdCsEPmGEQ0j7k8y > 9dLmmTBawDdckaxiGf1M1M/5R+f7XqU4e02Dnhk9v4umysmu5EH+VQFO4kfi5qX+ > DvaoS13lluyN1LGMyuwHkYFHXxlgZhR/XAW9Cn/0D/NDCQks6lHG9Sby+T/kvLSO > YNSBV78SX9wsJ/fH9MxrcTarxyUYsxnHdNkcarkmcrQIkhda/vBXQobXpYVL9poR > HmxNGNOujCTojqPJzlpgAaUVUq6pEUcN3u1vkCqXNQGVo983PHz2JhfEBGYihKBk > icbuFxfIy2Dv6KwuPPMV9Y7nVRNVBrVEpieZuAc7Sj9/U3kUgPUznUSbRFUCnGd1 > YI/kq9rn/qj/aqhaJdMF0Ci4+ieJMorKNUE1PWOjbnya8XoyBxXfPd5iV6n7elEw > V6qNyO6n46isrAiVEu5c > =MyIX > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
