Chris,
Top-posting is a post after another one I'm assuming? And sorry if it
wasn't related to Tomcat, I was just excited to finally making a bit of
headway on this issue.
Here is a full thread dump, sorry in advance if this doesn't follow the
etiquette on posting thread dumps:
"TP-Processor396" daemon prio=10 tid=0x00002aff2ba9d000 nid=0x5a7b waiting
on condition [0x00002aff61e98000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002afecfb91da0> (a
java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
at
com.tc.object.locks.LockStateNode$PendingLockHold.park(LockStateNode.java:179)
at
com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:723)
at
com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:701)
at com.tc.object.locks.ClientLockImpl.lock(ClientLockImpl.java:52)
at
com.tc.object.locks.ClientLockManagerImpl.lock(ClientLockManagerImpl.java:98)
at com.tc.object.bytecode.ManagerImpl.lock(ManagerImpl.java:747)
at
com.tc.object.bytecode.ManagerUtilInternal.beginLock(ManagerUtilInternal.java:33)
at
org.terracotta.locking.strategy.LongLockStrategy.beginLock(LongLockStrategy.java:16)
at
org.terracotta.locking.strategy.LongLockStrategy.beginLock(LongLockStrategy.java:7)
at
com.terracotta.toolkit.collections.ConcurrentDistributedMapDso.beginLock(ConcurrentDistributedMapDso.java:828)
at
com.terracotta.toolkit.collections.ConcurrentDistributedMapDso.get(ConcurrentDistributedMapDso.java:195)
at
com.terracotta.toolkit.collections.ConcurrentDistributedMapDsoArray.get(ConcurrentDistributedMapDsoArray.java:175)
at
org.terracotta.collections.ConcurrentDistributedMap.get(ConcurrentDistributedMap.java:190)
at
org.terracotta.cache.TerracottaDistributedCache.getNonExpiredEntry(TerracottaDistributedCache.java:197)
at
org.terracotta.cache.TerracottaDistributedCache.getNonExpiredEntryCoherent(TerracottaDistributedCache.java:131)
at
org.terracotta.cache.TerracottaDistributedCache.containsKey(TerracottaDistributedCache.java:126)
at
org.terracotta.modules.ehcache.store.ClusteredStore.internalContainsKey(ClusteredStore.java:476)
at
org.terracotta.modules.ehcache.store.ClusteredStore.containsKey(ClusteredStore.java:456)
at
org.terracotta.modules.ehcache.store.ClusteredStore.containsKeyInMemory(ClusteredStore.java:463)
at net.sf.ehcache.Cache.searchInStoreWithStats(Cache.java:1742)
at net.sf.ehcache.Cache.get(Cache.java:1405)
at
net.sf.ehcache.hibernate.regions.EhcacheTransactionalDataRegion.get(EhcacheTransactionalDataRegion.java:91)
at
net.sf.ehcache.hibernate.strategy.AbstractReadWriteEhcacheAccessStrategy.get(AbstractReadWriteEhcacheAccessStrategy.java:65)
at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:524)
at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:397)
at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:219)
at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:443)
at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:119)
at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:105)
at
org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:587)
at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:542)
at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:397)
at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:219)
at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:443)
at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:119)
at
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:105)
at
org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:587)
at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:542)
at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:397)
at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
at org.hibernate.type.EntityType.resolve(EntityType.java:412)
at
org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:139)
at
org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
at org.hibernate.loader.Loader.doQuery(Loader.java:752)
at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.loadEntity(Loader.java:1885)
at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:71)
at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:65)
at
org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3062)
at
org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:434)
at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:415)
at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:874)
at
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:590)
at org.hibernate.type.EntityType.resolve(EntityType.java:412)
at
org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:139)
at
org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
at org.hibernate.loader.Loader.doQuery(Loader.java:752)
at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.loadEntity(Loader.java:1885)
at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:71)
at
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:65)
at
org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3062)
at
org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:434)
at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:415)
at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:165)
at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:223)
at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:906)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:843)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:836)
at
org.springframework.orm.hibernate3.HibernateTemplate$1.doInHibernate(HibernateTemplate.java:528)
at
org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
at
org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at
org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:522)
at
org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:516)
at
com.thelearningbar.persistence.hibernate.SurveyParticipantDaoImpl.getSurveyParticipantById(SurveyParticipantDaoImpl.java:243)
at
com.thelearningbar.service.impl.SurveyParticipantManagerImpl.getSurveyParticipantById(SurveyParticipantManagerImpl.java:306)
at sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy20.getSurveyParticipantById(Unknown Source)
at
com.thelearningbar.web.SurveyFlowController.handleRequest(SurveyFlowController.java:187)
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:523)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:463)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)
at
org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
at
org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at
org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at
org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at
org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at
org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
at
org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.tcInvoke(SessionValve55.java:99)
at
org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.invoke(SessionValve55.java:86)
at
org.terracotta.session.ModernTomcatSessionValve.invoke(ModernTomcatSessionValve.java:66)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
Cheers,
Charles
On Wed, May 8, 2013 at 9:42 AM, Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Charles,
>
> On 5/8/13 8:31 AM, Charles Richard wrote:
> > On Wed, May 8, 2013 at 9:27 AM, Daniel Mikusa
> > <dmik...@gopivotal.com> wrote:
> >> On May 8, 2013, at 8:20 AM, Charles Richard wrote:
> >>
> >>> Hi,
> >>>
> >>> We have a weird issue on our site which some random trigger
> >>> event will backup all c3p0 connections until it hits the max
> >>> pool size.
> >>>
> >>> I have scripts that will do a softReset on the c3p0 connection
> >>> pool when they hit their max so help us manage the issue and to
> >>> also help me have time to hopefully get some decent thread
> >>> dumps to catch the underlying issue.
> >>>
> >>> The problem happened yesterday and I get a lot of these:
> >>>
> >>> "TP-Processor396" daemon prio=10 tid=0x00002aff2ba9d000
> >>> nid=0x5a7b
> >> waiting
> >>> on condition [0x00002aff61e98000] java.lang.Thread.State:
> >>> WAITING (parking) at sun.misc.Unsafe.park(Native Method) -
> >>> parking to wait for <0x00002afecfb91da0> (a
> >>> java.util.concurrent.Semaphore$NonfairSync) at
> >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >>>
> >>
> at
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> >>>
> >>
> at
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> >>>
> >>
> at
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> >>>
> >>
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
> >>> at
> >>>
> >>
> com.tc.object.locks.LockStateNode$PendingLockHold.park(LockStateNode.java:179)
> >>>
> >>
> at
> >>>
> com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:723)
> >>>
> >>>
> at
> >>>
> com.tc.object.locks.ClientLockImpl.acquireQueued(ClientLockImpl.java:701)
> >>>
> >>>
> at com.tc.object.locks.ClientLockImpl.lock(ClientLockImpl.java:52)
> >>> at
> >>>
> >>
> com.tc.object.locks.ClientLockManagerImpl.lock(ClientLockManagerImpl.java:98)
> >>>
> >>
> at com.tc.object.bytecode.ManagerImpl.lock(ManagerImpl.java:747)
> >>
> >>>
> >>> If I look at a stack before the issue happened, I see no
> >>> TP-Processor threads with the "parking to wait for". What can i
> >>> read into this?
>
> >> What in your application is using the "com.tc.object.locks"
> >> package? This is not used by Tomcat.
> >
> > We are using Terracotta which is a bit of a black box to me (setup
> > I've inherited). Terracotta helps us with the Tomcat sessions being
> > "transportable" across front end servers.
>
> Please don't top-post: it's confusing and hard to follow.
>
> Unfortunately, I don't see any Tomcat code in here anywhere, so I
> can't really help (yet). Is that the whole stack trace? I find it
> difficult to believe there "bytecode.ManagerImpl" is a Runnable or
> extends Thread, especially because the "run" method isn't the lowest
> in the stack trace.
>
> Can you give us more information?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJRikhNAAoJEBzwKT+lPKRY53QP/1oz2eR+CRG+g/+2Oedh1UjE
> FckiVRQFkLsFSJBelGdQU+x0wBljgiSiOUgVaEQotpnaLO8pQCFmjMPcCsAZ2YiY
> xmnfHf2s89w/YGXo7HNB5X1okhpn+1pzjznmUvHGLGKO5+OUTdJvZ2stS6m6uEfg
> Vtmn6IQZywUyLKOcZ/Tl/UESirYjIaT5ky3Mvxbb/9JAyXICutJtxZQdZDG7U1du
> UU7BwmMQP/7ZiCiR9zUoarr2MT5FTgy1mpBGGI0OopmZilvo83UiyBcJJ/lN5miz
> PlE0BqisOeZIk5hAX+FqUZ1rTFvjpPNQjfN/9TRelhOs9fP9ZaTQ4PIkjwP8h2R7
> V2aqQmIrrZSrjS2SRsPx+iEnd6klwGpvlPoCyuWhikjKdv06nwZ7iDLWePw9lek7
> x8q6KeuE5KUy5w0EGN23zx54sh92eipjXj1uBBWBoA8cA6DZEXEPG+cGGoj/cR9+
> 9mhdOUbKSd9MQUR4H1b2+sErTZhj1KeBPxnBxeUU7FVoIn7J/x+DMXtYMU5OX6Kq
> jO0YTAXUMGAb6WPR2pz5bU2nvlev8JVVDwtjmV6l3P+czAqVQ7aUDc6aRfPbcM+D
> RJb4i64Ww41ZgbJv4YQ5SkxgdLIcjlzMu07pPXUwonlQI8NGvZsMN60eeKXnrrX5
> tnst+5jt/p/7mSULqAtR
> =ATll
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
