I haven't had a chance to run a load test of j6u1 vs j6u2 with 5.0.5, however switching to j6u2 on our public test site for the last day seems to have solved the problem. The previous day on j6u1 it stopped responding 4 times. Thanks for pointing out the bug report.
-Matt -----Original Message----- From: Ben Sommerville [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 10, 2007 10:52 PM To: 'Tapestry users' Subject: RE: T5: NPE in Base64InputStream and locked/waitingCheckForUpdatesFilter If you are running under jdk 6u1 and tapestry 5.0.5 (or greater) then there is a jvm bug that can cause a deadlock. The bug report is at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6571733. It is supposed to be fixed in jdk 6u2 (which was release recently) but I haven't had a chance to test it yet. cheers Ben > -----Original Message----- > From: Martin Grotzke [mailto:[EMAIL PROTECTED] > Sent: Wednesday, 11 July 2007 9:06 AM > To: Tapestry users > Subject: Re: T5: NPE in Base64InputStream and > locked/waitingCheckForUpdatesFilter > > The NPE seems to be caused by a missing t:formdata request parameter. > > I just "reproduced" this (weird) situation by removing the hidden form > field "t:formdata" with firebug, so that I could produce the NPE. > > Although, following request went through well, and the number > of request > processing threads did not increase. > > So there seems to be no (direct) interrelationship between the NPE > and the locked threads. > > Cheers, > Martin > > > > On Tue, 2007-07-10 at 23:43 +0200, Martin Grotzke wrote: > > Hi, > > > > we had an issue with our deployed application that did not respond > > anymore. This happened two or three times in the last 4 days, but > > I was not able to reproduce it until now. > > > > The analysis of the logs showed, that there was a NPE in > > Base64InputStream, and afterwards the application did not respond > > anymore. > > > > When I triggered a thread dump, all 200 tomcat threads were > in status > > WAITING, like this one: > > > > "http-9090-1" daemon prio=10 tid=0x00002aaaf7e1fc00 > nid=0x3f05 waiting on condition > [0x000000004459e000..0x000000004459fbc0] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab8228360> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndC > heckInterrupt(AbstractQueuedSynchronizer.java:712) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquir > eShared(AbstractQueuedSynchronizer.java:842) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireS > hared(AbstractQueuedSynchronizer.java:1162) > > at > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.loc k(ReentrantReadWriteLock.java:594) > > at > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > ad(ConcurrentBarrier.java:70) > > at > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > rvice(CheckForUpdatesFilter.java:110) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > at > org.apache.tapestry.services.TapestryModule$11.service(Tapestr yModule.java:1044) > > at > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq uestHandler_1139c29ae40.java) > > at > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > er(ApplicationFilterChain.java:235) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > cationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > rapperValve.java:230) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardC > ontextValve.java:175) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > Valve.java:128) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > Valve.java:104) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > gineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > ter.java:261) > > at > org.apache.coyote.http11.Http11Processor.process(Http11Process > or.java:844) > > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle r.process(Http11Protocol.java:581) > > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint. java:447) > > at java.lang.Thread.run(Thread.java:619) > > > > I'm not sure if the NPE that happened before is the reason > for this, as I don't > > see the relationship between both issues (apart from the > correlation of events/time). > > > > Hopefully someone closer to the code can have a look at > this and tell what's the > > problem here? Then I'd like to submit a bug report / help > fixing(?)... > > > > More output of the logs you find at the end of this email. > > > > Thanx a lot in advance, > > cheers, > > Martin > > > > > > > > [INFO ] 2007-07-09 11:56:35,513 TP-Processor1 > org.comp.proj.search.presentation.pages.Search.onActivate: > > Got invoked args hose, cat:Herrenbekleidung, n, 100 > > > > [INFO ] 2007-07-09 11:56:35,518 TP-Processor1 > org.comp.proj.search.presentation.pages.Search.setupRender: > > Starting search... > > > > [DEBUG] 2007-07-09 11:56:35,520 TP-Processor1 > org.comp.proj.search.business.SearchServiceSolrImpl.search: > > Starting query: > > > q=hose;score+desc&fq=cat:Herrenbekleidung&start=0&rows=100&fac et=true&facet.field=cat&facet.field=brand&facet.field=type&facet> .field=color&facet > > > .field=price&facet.limit=5&facet.zeros=false&facet.missing=fal se&version=2.2 > > > > [DEBUG] 2007-07-09 11:56:35,610 TP-Processor1 > org.comp.proj.search.business.SearchServiceSolrImpl.logResponse: > > Got response: numFound: 207, queryTime: 43 > > > > [INFO ] 2007-07-09 11:56:35,803 TP-Processor1 > org.comp.proj.search.presentation.services.AppModule.TimingFil > ter.service: > > Request time: 292 ms > > > > [INFO ] 2007-07-09 11:57:05,982 TP-Processor5 > org.comp.proj.search.presentation.pages.Search.onActivate: > > Got invoked args hose, n, 100 > > > > [INFO ] 2007-07-09 11:57:05,988 TP-Processor5 > org.comp.proj.search.presentation.pages.Search.setupRender: > > Starting search... > > > > [DEBUG] 2007-07-09 11:57:05,989 TP-Processor5 > org.comp.proj.search.business.SearchServiceSolrImpl.search: > > Starting query: > > > q=hose;score+desc&start=0&rows=100&facet=true&facet.field=cat& facet.field=brand&facet.field=type&facet.field=color&facet.field> =price&facet.limit > > =5&facet.zeros=false&facet.missing=false&version=2.2 > > > > [DEBUG] 2007-07-09 11:57:06,078 TP-Processor5 > org.comp.proj.search.business.SearchServiceSolrImpl.logResponse: > > Got response: numFound: 2001, queryTime: 44 > > > > [INFO ] 2007-07-09 11:57:06,276 TP-Processor5 > org.comp.proj.search.presentation.services.AppModule.TimingFil > ter.service: > > Request time: 296 ms > > > > [INFO ] 2007-07-09 12:07:22,379 TP-Processor5 > org.comp.proj.search.presentation.pages.Search.onActivate: > > Got invoked args > > > > [INFO ] 2007-07-09 12:07:22,382 TP-Processor5 > org.comp.proj.search.presentation.services.AppModule.TimingFil > ter.service: > > Request time: 605041 ms > > ^^^^^^^^^^^^^^^^^^^^^^^ --> 10 minutes?! > > > > [ERROR] 2007-07-09 12:07:22,382 TP-Processor5 > org.apache.tapestry.internal.services.InternalModule.RequestEx > ceptionHandler.handleRequestExceptio > > n: > > Processing of request failed with uncaught exception: > org.apache.tapestry.ioc.internal.util.TapestryException > > > > org.apache.tapestry.ioc.internal.util.TapestryException > > at > org.apache.tapestry.corelib.components.Form.onAction(Form.java:375) > > at > org.apache.tapestry.corelib.components.Form.handleComponentEve > nt(Form.java) > > at > org.apache.tapestry.internal.structure.ComponentPageElementImp > l.handleEvent(ComponentPageElementImpl.java:893) > > at > org.apache.tapestry.internal.structure.ComponentPageElementImp > l.triggerEvent(ComponentPageElementImpl.java:998) > > at > org.apache.tapestry.internal.services.ComponentActionRequestHa > ndlerImpl.handle(ComponentActionRequestHandlerImpl.java:81) > > at > org.apache.tapestry.internal.services.InternalModule$11.handle (InternalModule.java:541) > > at > $ComponentActionRequestHandler_1139c29ae5a.handle($ComponentAc tionRequestHandler_1139c29ae5a.java) > > at > $ComponentActionRequestHandler_1139c29ae55.handle($ComponentAc tionRequestHandler_1139c29ae55.java) > > at > org.apache.tapestry.internal.services.ComponentActionDispatche > r.dispatch(ComponentActionDispatcher.java:116) > > at > $Dispatcher_1139c29ae57.dispatch($Dispatcher_1139c29ae57.java) > > at > $Dispatcher_1139c29ae49.dispatch($Dispatcher_1139c29ae49.java) > > at > org.apache.tapestry.services.TapestryModule$12.service(Tapestr yModule.java:1066) > > at > org.comp.proj.search.presentation.services.AppModule$2.service (AppModule.java:117) > > at > $RequestFilter_1139c29ae48.service($RequestFilter_1139c29ae48.java) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > org.comp.proj.search.presentation.services.AppModule$1.service (AppModule.java:76) > > at > $RequestFilter_1139c29ae47.service($RequestFilter_1139c29ae47.java) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > org.apache.tapestry.internal.services.LocalizationFilter.servi > ce(LocalizationFilter.java:43) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > org.apache.tapestry.services.TapestryModule$2.service(Tapestry Module.java:657) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > org.apache.tapestry.internal.services.StaticFilesFilter.servic > e(StaticFilesFilter.java:63) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > org.apache.tapestry.internal.services.CheckForUpdatesFilter$2. invoke(CheckForUpdatesFilter.java:97) > > at > org.apache.tapestry.internal.services.CheckForUpdatesFilter$2. invoke(CheckForUpdatesFilter.java:88) > > at > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > ad(ConcurrentBarrier.java:77) > > at > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > rvice(CheckForUpdatesFilter.java:110) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > at > org.apache.tapestry.services.TapestryModule$11.service(Tapestr yModule.java:1044) > > at > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq uestHandler_1139c29ae40.java) > > at > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > er(ApplicationFilterChain.java:235) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > cationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > rapperValve.java:230) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardC > ontextValve.java:175) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > Valve.java:128) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > Valve.java:104) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > gineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > ter.java:261) > > at > org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) > > at > org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) > > at > org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767) > > at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSo > cket.java:697) > > at > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(Chan nelSocket.java:889) > > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( ThreadPool.java:686) > > at java.lang.Thread.run(Thread.java:619) > > Caused by: java.lang.NullPointerException > > at > org.apache.tapestry.internal.util.Base64InputStream.decode(Bas > e64InputStream.java:34) > > at > org.apache.tapestry.internal.util.Base64InputStream.<init>(Bas > e64InputStream.java:29) > > at > org.apache.tapestry.internal.util.Base64ObjectInputStream.<ini > t>(Base64ObjectInputStream.java:30) > > at > org.apache.tapestry.corelib.components.Form.onAction(Form.java:355) > > ... 47 more > > Jul 9, 2007 2:27:57 PM > org.apache.tomcat.util.threads.ThreadPool logFull > > SEVERE: All threads (200) are currently busy, waiting. > Increase maxThreads (200) or check the servlet status > > 2007-07-09 17:13:35 > > Full thread dump Java HotSpot(TM) 64-Bit Server VM > (1.6.0_01-b06 mixed mode): > > > > "http-9090-1" daemon prio=10 tid=0x00002aaaf7e1fc00 > nid=0x3f05 waiting on condition > [0x000000004459e000..0x000000004459fbc0] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab8228360> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndC > heckInterrupt(AbstractQueuedSynchronizer.java:712) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquir > eShared(AbstractQueuedSynchronizer.java:842) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireS > hared(AbstractQueuedSynchronizer.java:1162) > > at > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.loc k(ReentrantReadWriteLock.java:594) > > at > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > ad(ConcurrentBarrier.java:70) > > at > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > rvice(CheckForUpdatesFilter.java:110) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > at > org.apache.tapestry.services.TapestryModule$11.service(Tapestr yModule.java:1044) > > at > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq uestHandler_1139c29ae40.java) > > at > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > er(ApplicationFilterChain.java:235) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > cationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > rapperValve.java:230) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardC > ontextValve.java:175) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > Valve.java:128) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > Valve.java:104) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > gineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > ter.java:261) > > at > org.apache.coyote.http11.Http11Processor.process(Http11Process > or.java:844) > > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle r.process(Http11Protocol.java:581) > > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint. java:447) > > at java.lang.Thread.run(Thread.java:619) > > > > "TP-Processor200" daemon prio=10 tid=0x00002aaaf8896400 > nid=0x3ce9 waiting on condition > [0x000000004455d000..0x000000004455ec40] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab8228360> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndC > heckInterrupt(AbstractQueuedSynchronizer.java:712) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquir > eShared(AbstractQueuedSynchronizer.java:842) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireS > hared(AbstractQueuedSynchronizer.java:1162) > > at > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.loc k(ReentrantReadWriteLock.java:594) > > at > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > ad(ConcurrentBarrier.java:70) > > at > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > rvice(CheckForUpdatesFilter.java:110) > > at > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > at > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > at > org.apache.tapestry.services.TapestryModule$11.service(Tapestr yModule.java:1044) > > at > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq uestHandler_1139c29ae40.java) > > at > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > er(ApplicationFilterChain.java:235) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > cationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > rapperValve.java:230) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardC > ontextValve.java:175) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > Valve.java:128) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > Valve.java:104) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > gineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > ter.java:261) > > at > org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) > > at > org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) > > at > org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767) > > at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSo > cket.java:697) > > at > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(Chan nelSocket.java:889) > > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( ThreadPool.java:686) > > at java.lang.Thread.run(Thread.java:619) > > > > ... and 199 more of them > > > > > -- > Martin Grotzke > http://www.javakaffee.de/blog/ > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
