On Wed, May 8, 2013 at 2:33 PM, Christopher Schultz < ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Chuck, > > On 5/8/13 11:25 AM, Caldarale, Charles R wrote: > >> From: Charles Richard [mailto:charle...@thelearningbar.com] > >> Subject: Re: Tomcat thread dump analysis > > > >> Here is a full thread dump > > > > Which again shows no Tomcat involvement in the locking hang. > > +1 > > At least it confirms OP is actually running Tomcat, which wasn't > evident from the first stack trace. > > Charles, please understand that what you have presented is a stack > trace, not a thread dump: the difference is that a thread dump is a > stack trace for every live thread in the JVM (which can show > deadlocks) and you have provided only a single stack trace which > cannot show a deadlock (it can only show that the thread is waiting on > some condition). > > To demonstrate deadlock, you need to be able to see the condition > being awaited in one thread, a lock held in another thread, and then > the opposite scenario in another thread (or a single thread running > into its own locks). Your stack trace does not show that the thread > already holds the lock being awaited at the top of the stack trace. > > I appreciate the friendly feedback! How do I show a lock? I don't see any threads that have a "BLOCKED" status. I do get this when I do a grep: [root@web01 stacks]# grep locked tomcat1_20130507_14\:38.stack | wc -l 154 My connection pool is 150 and I do see 150 locks in the following type scenario which has a "locked": "TP-Processor416" daemon prio=10 tid=0x00002aff2939a800 nid=0x5ae7 in Object.wait() [0x00002aff632ae000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002afe240fbc58> (a com.mchange.v2.resourcepool.BasicResourcePool) at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315) at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557) - locked <0x00002afe240fbc58> (a com.mchange.v2.resourcepool.BasicResourcePool) > It is entirely possible that both Spring and Terrecotta attempt to > synchronize on the same object (HttpSession) and that the thread is > deadlocking on itself. If this is trivially-repeatable every time > Terrecotta attempts to do whatever it is doing (hard to tell... > Hibernate is executing a SQL query which ... uses the session?!?), > then perhaps you have identified the problem and the Spring > configuration to synchronize on a session attribute rather than the > session itself will fix the problem. > > On the other hand, if the problem is different and yet you have > arrived at your "Spring + Terrecotta = deadlock" conclusion due to > treating Google as a consultant, then you may be barking up the wrong > tree. > > Sorry: we like to get enough facts before we tell you a) what the > problem is and b) how to fix it rather than just saying "I Googled for > 'terra cotta deadlock' and someone said that if you use Spring, you > get deadlocks" and telling you to go away. Instead, we'd rather help > you fix your actual problem and educate you at the same time. > > I agree. At times desperate times call for desperate actions when I've used all existing knowledge, Google can give me ideas of where I could possibly look next. Given the choice, I would rather be educated on how to diagnose this issue properly. > It would appear that logic in your application threads has either > > created a deadlock, or failed to unlock something before > > returning, > > That's a tall order unless native code is involved, I believe. Though > I don't have much experience with the java.lang.concurrent package... > > > or something else entirely (e.g, broken network causing grief with > > Terracotta's distributed caching). If it's a deadlock, you need > > to examine the entire thread dump (all threads) and look for a > > call stack that isn't like the others but holds the lock everyone > > else is waiting for. If it's simply a failure to unlock somewhere, > > you will likely have to do a thorough code review to find it. You > > will most likely need Terracotta's help here, since it's their code > > waiting on the lock. > > +1 > > Start with finding out what Terrecotta is trying to lock on: if it's > not the session, then you know you're entirely wrong about the cause. > > Also, knowing whether the thread is deadlocking itself versus another > thread deadlocking with it is a very important piece of information to > have. > > - -chris > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJRioxWAAoJEBzwKT+lPKRYz2AQAKbMT6qRh1oKvKdgBEUe1fJl > yqgyB0JOkb9pZ42P2bVovJJtO5A15tWw8NigEbKc+CwJMv9rgOzqRY7y30oih9mK > yhu5C/sEFDPlPPzt0H5CT3YFqVOoJMpstmgFBA0FBLKEaqmzdc61WIohQqJmv2J6 > 2z3vNvmH1xBucEqgOrOEn8mp9rbaOHVmQtf6Zh1QaxhBzA0CREnnwz0FvFEhklGx > rlpwkQafjcKjCCP6eaiurtZJtv4S0hJQJSOmLQKBdm3DY/dRokeN/ROnxn3DpqfX > e2JP4eAC2NYmKHwhXY3SOviK5Ha9NSB1fbBGSPSPQrjQ2In3UccYs6CENKOKUa68 > 0WwONHAw/OY0MzCJ6M9cZicX9XhNDNeRUzYlkORQpeAloM5w7Tp9q5fKo3iabFcC > NwuWN5O5H5TZ2MSH7UHV9Kf0EA1Zk7Howi2Ea2zV4EnzutdgM5yxq//ow8LTcOdi > ED376ckgSMgfV/b8u2tfJ1M9bD5WjD8v+kVbu+ErYzuM6i28iAGN7wrfgO+/Zn2S > BuQgcEq/3usUXwnQ5IBAYpVF5DAKv/GeAaV+C8Ft+mb+jgo00Hb7Eqn5eY/w6CL/ > 1DM0mgmfcHTlSP4JK5noV73Eip2LmxdPwDFvNBkIiRPq5Av+FSIfIKclJu/piMQ/ > d0SCj26GbqSEIOYVUYX7 > =3SsJ > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
