Hi, we had an issue with our deployed application that did not respond anymore. This happened two or three times in the last 4 days, but I was not able to reproduce it until now.
The analysis of the logs showed, that there was a NPE in
Base64InputStream, and afterwards the application did not respond
anymore.
When I triggered a thread dump, all 200 tomcat threads were in status
WAITING, like this one:
"http-9090-1" daemon prio=10 tid=0x00002aaaf7e1fc00 nid=0x3f05 waiting on
condition [0x000000004459e000..0x000000004459fbc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab8228360> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:712)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:842)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1162)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at
org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:70)
at
org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:110)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java)
at
org.apache.tapestry.services.TapestryModule$11.service(TapestryModule.java:1044)
at
$HttpServletRequestHandler_1139c29ae40.service($HttpServletRequestHandler_1139c29ae40.java)
at org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
I'm not sure if the NPE that happened before is the reason for this, as I don't
see the relationship between both issues (apart from the correlation of
events/time).
Hopefully someone closer to the code can have a look at this and tell what's the
problem here? Then I'd like to submit a bug report / help fixing(?)...
More output of the logs you find at the end of this email.
Thanx a lot in advance,
cheers,
Martin
[INFO ] 2007-07-09 11:56:35,513 TP-Processor1
org.comp.proj.search.presentation.pages.Search.onActivate:
Got invoked args hose, cat:Herrenbekleidung, n, 100
[INFO ] 2007-07-09 11:56:35,518 TP-Processor1
org.comp.proj.search.presentation.pages.Search.setupRender:
Starting search...
[DEBUG] 2007-07-09 11:56:35,520 TP-Processor1
org.comp.proj.search.business.SearchServiceSolrImpl.search:
Starting query:
q=hose;score+desc&fq=cat:Herrenbekleidung&start=0&rows=100&facet=true&facet.field=cat&facet.field=brand&facet.field=type&facet.field=color&facet
.field=price&facet.limit=5&facet.zeros=false&facet.missing=false&version=2.2
[DEBUG] 2007-07-09 11:56:35,610 TP-Processor1
org.comp.proj.search.business.SearchServiceSolrImpl.logResponse:
Got response: numFound: 207, queryTime: 43
[INFO ] 2007-07-09 11:56:35,803 TP-Processor1
org.comp.proj.search.presentation.services.AppModule.TimingFilter.service:
Request time: 292 ms
[INFO ] 2007-07-09 11:57:05,982 TP-Processor5
org.comp.proj.search.presentation.pages.Search.onActivate:
Got invoked args hose, n, 100
[INFO ] 2007-07-09 11:57:05,988 TP-Processor5
org.comp.proj.search.presentation.pages.Search.setupRender:
Starting search...
[DEBUG] 2007-07-09 11:57:05,989 TP-Processor5
org.comp.proj.search.business.SearchServiceSolrImpl.search:
Starting query:
q=hose;score+desc&start=0&rows=100&facet=true&facet.field=cat&facet.field=brand&facet.field=type&facet.field=color&facet.field=price&facet.limit
=5&facet.zeros=false&facet.missing=false&version=2.2
[DEBUG] 2007-07-09 11:57:06,078 TP-Processor5
org.comp.proj.search.business.SearchServiceSolrImpl.logResponse:
Got response: numFound: 2001, queryTime: 44
[INFO ] 2007-07-09 11:57:06,276 TP-Processor5
org.comp.proj.search.presentation.services.AppModule.TimingFilter.service:
Request time: 296 ms
[INFO ] 2007-07-09 12:07:22,379 TP-Processor5
org.comp.proj.search.presentation.pages.Search.onActivate:
Got invoked args
[INFO ] 2007-07-09 12:07:22,382 TP-Processor5
org.comp.proj.search.presentation.services.AppModule.TimingFilter.service:
Request time: 605041 ms
^^^^^^^^^^^^^^^^^^^^^^^ --> 10 minutes?!
[ERROR] 2007-07-09 12:07:22,382 TP-Processor5
org.apache.tapestry.internal.services.InternalModule.RequestExceptionHandler.handleRequestExceptio
n:
Processing of request failed with uncaught exception:
org.apache.tapestry.ioc.internal.util.TapestryException
org.apache.tapestry.ioc.internal.util.TapestryException
at org.apache.tapestry.corelib.components.Form.onAction(Form.java:375)
at
org.apache.tapestry.corelib.components.Form.handleComponentEvent(Form.java)
at
org.apache.tapestry.internal.structure.ComponentPageElementImpl.handleEvent(ComponentPageElementImpl.java:893)
at
org.apache.tapestry.internal.structure.ComponentPageElementImpl.triggerEvent(ComponentPageElementImpl.java:998)
at
org.apache.tapestry.internal.services.ComponentActionRequestHandlerImpl.handle(ComponentActionRequestHandlerImpl.java:81)
at
org.apache.tapestry.internal.services.InternalModule$11.handle(InternalModule.java:541)
at
$ComponentActionRequestHandler_1139c29ae5a.handle($ComponentActionRequestHandler_1139c29ae5a.java)
at
$ComponentActionRequestHandler_1139c29ae55.handle($ComponentActionRequestHandler_1139c29ae55.java)
at
org.apache.tapestry.internal.services.ComponentActionDispatcher.dispatch(ComponentActionDispatcher.java:116)
at $Dispatcher_1139c29ae57.dispatch($Dispatcher_1139c29ae57.java)
at $Dispatcher_1139c29ae49.dispatch($Dispatcher_1139c29ae49.java)
at
org.apache.tapestry.services.TapestryModule$12.service(TapestryModule.java:1066)
at
org.comp.proj.search.presentation.services.AppModule$2.service(AppModule.java:117)
at $RequestFilter_1139c29ae48.service($RequestFilter_1139c29ae48.java)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at
org.comp.proj.search.presentation.services.AppModule$1.service(AppModule.java:76)
at $RequestFilter_1139c29ae47.service($RequestFilter_1139c29ae47.java)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at
org.apache.tapestry.internal.services.LocalizationFilter.service(LocalizationFilter.java:43)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at
org.apache.tapestry.services.TapestryModule$2.service(TapestryModule.java:657)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at
org.apache.tapestry.internal.services.StaticFilesFilter.service(StaticFilesFilter.java:63)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at
org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:97)
at
org.apache.tapestry.internal.services.CheckForUpdatesFilter$2.invoke(CheckForUpdatesFilter.java:88)
at
org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:77)
at
org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:110)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java)
at
org.apache.tapestry.services.TapestryModule$11.service(TapestryModule.java:1044)
at
$HttpServletRequestHandler_1139c29ae40.service($HttpServletRequestHandler_1139c29ae40.java)
at org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
at
org.apache.tapestry.internal.util.Base64InputStream.decode(Base64InputStream.java:34)
at
org.apache.tapestry.internal.util.Base64InputStream.<init>(Base64InputStream.java:29)
at
org.apache.tapestry.internal.util.Base64ObjectInputStream.<init>(Base64ObjectInputStream.java:30)
at org.apache.tapestry.corelib.components.Form.onAction(Form.java:355)
... 47 more
Jul 9, 2007 2:27:57 PM org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads
(200) or check the servlet status
2007-07-09 17:13:35
Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.6.0_01-b06 mixed mode):
"http-9090-1" daemon prio=10 tid=0x00002aaaf7e1fc00 nid=0x3f05 waiting on
condition [0x000000004459e000..0x000000004459fbc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab8228360> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:712)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:842)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1162)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at
org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:70)
at
org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:110)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java)
at
org.apache.tapestry.services.TapestryModule$11.service(TapestryModule.java:1044)
at
$HttpServletRequestHandler_1139c29ae40.service($HttpServletRequestHandler_1139c29ae40.java)
at org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
"TP-Processor200" daemon prio=10 tid=0x00002aaaf8896400 nid=0x3ce9 waiting on
condition [0x000000004455d000..0x000000004455ec40]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab8228360> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:712)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:842)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1162)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at
org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRead(ConcurrentBarrier.java:70)
at
org.apache.tapestry.internal.services.CheckForUpdatesFilter.service(CheckForUpdatesFilter.java:110)
at $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java)
at $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java)
at
org.apache.tapestry.services.TapestryModule$11.service(TapestryModule.java:1044)
at
$HttpServletRequestHandler_1139c29ae40.service($HttpServletRequestHandler_1139c29ae40.java)
at org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
at java.lang.Thread.run(Thread.java:619)
... and 199 more of them
signature.asc
Description: This is a digitally signed message part
