I really appreciate your input Leon.
On Wed, 2009-01-28 at 11:07 +0100, Leon Rosenberg wrote:
> > "RMI TCP Connection(42)-173.x.x.x" - Thread t...@112
> > java.lang.Thread.State: RUNNABLE
>
> reading from socket, usually not a problem.
>
I thought so. Thanks.
> > "http-8081-35" - Thread t...@71
> > java.lang.Thread.State: RUNNABLE
> > at java.lang.String.equals(String.java:1018)
> > at
> > com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:554)
> > at
> > org.codehaus.xfire.soap.handler.ReadHeadersHandler.invoke(ReadHeadersHandler.java:44)
> This one actually looks like a problem, was the server under load as
> you created the dump, or did you removed the load first?
No, the server was not under load at all. And I took various thread
dumps at random intervals, and they thread was still there (same thread
number). Maybe there is an infinite loop, but I don't know why it works
for a random amount of time, and then freezes.
> If there were no external requests to the server, you most probably
> have an infinite loop somewhere (how stable is xfire anyway?) or you
> have a really really really large xml file to parse.
That's a good question. I have no experience with XFire. Would there be
a way to know which XML file it's parsing?
>
> > "http-8081-31" - Thread t...@67
> > java.lang.Thread.State: RUNNABLE
> > at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl
> > $PrologDriver.next(XMLDocumentScannerImpl.java:930)
> > at
> > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
> > at
> > com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
> > at
> > com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:548)
> > at
> > org.codehaus.xfire.soap.handler.ReadHeadersHandler.invoke(ReadHeadersHandler.java:44)
> > at
> > org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
> > at
> > org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
> > at
> > org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
> > at
> > org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)
> > at
> > org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)
> > at
> > org.codehaus.xfire.transport.http.XFireServlet.doPost(XFireServlet.java:116)
> >
>
> note that the XFire path is almost identical.
> Personally I neved managed to catch healthy code in something like
> String.equals(), but it may be possible, even imho unlikeable.
> The dump looks healthy, but is it the dump from the tomcat 5.5 or tomcat 6?
> Because if tomcat 6 is the problem and the dump is from tomcat 5 its useless
> :-)
>
This was the heap dump from Tomcat 6.
> you need more research :-) For now it looks like your first tomcat is
> waiting for the second tomcat which just can't handle the requests as
> fast as its expected. You may want to check configuration options of
> both (threads in the connector etc).
If the threadMax would be too low in the connector, wouldn't the
"freeze" be over once there are free connections? And also, how can a
small threadMax make a thread hang? For example the one that is trying
to read an XML file.
> Also the heap dump of the second tomcat would be useful (if this one
> is from first).
> And a thread dump shortly after the traffic has gone (for example kill
> first tomcat, wait 1 minute, make thread dump).
>
Did this, see the output (I removed the RMI threads):
"TP-Monitor" - Thread t...@28
java.lang.Thread.State: TIMED_WAITING on
org.apache.tomcat.util.threads.threadpool$monitorrunna...@498364
at java.lang.Object.wait(Native Method)
at
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"TP-Processor4" - Thread t...@27
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked java.net.sockssocketi...@134af1
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:306)
at
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:660)
at
org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:870)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"TP-Processor3" - Thread t...@26
java.lang.Thread.State: WAITING on
org.apache.tomcat.util.threads.threadpool$controlrunna...@514577
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"TP-Processor2" - Thread t...@25
java.lang.Thread.State: WAITING on
org.apache.tomcat.util.threads.threadpool$controlrunna...@1a79b48
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"TP-Processor1" - Thread t...@24
java.lang.Thread.State: WAITING on
org.apache.tomcat.util.threads.threadpool$controlrunna...@c3a53b
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"http-8081-Acceptor-0" - Thread t...@23
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked java.net.sockssocketi...@9d478b
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t...@22
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"Monitor Runner" - Thread t...@21
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.icesoft.util.MonitorRunner$1.run(MonitorRunner.java:20)
Locked ownable synchronizers:
- None
"pool-1-thread-1" - Thread t...@20
java.lang.Thread.State: TIMED_WAITING on
java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@94c5e7
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.StandardQueryCache Spool Thread" - Thread t...@18
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at
net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:563)
at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:64)
at
net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1047)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.UpdateTimestampsCache Spool Thread" - Thread t...@17
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at
net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:563)
at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:64)
at
net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1047)
Locked ownable synchronizers:
- None
"HSQLDB Timer @55d1b4" - Thread t...@16
java.lang.Thread.State: TIMED_WAITING on
org.hsqldb.lib.hsqltimer$taskqu...@14cff90
at java.lang.Object.wait(Native Method)
at org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)
at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"Timer-0" - Thread t...@15
java.lang.Thread.State: WAITING on java.util.taskqu...@1e6661
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
Locked ownable synchronizers:
- None
"Session Monitor" - Thread t...@14
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at
com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Listener$1.run(SessionDispatcher.java:208)
Locked ownable synchronizers:
- None
"Store userCache Spool Thread" - Thread t...@13
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at
net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:573)
at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
at
net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1057)
Locked ownable synchronizers:
- None
"Signal Dispatcher" - Thread t...@4
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" - Thread t...@3
java.lang.Thread.State: WAITING on java.lang.ref.referencequeue$l...@1d36f77
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Locked ownable synchronizers:
- None
"Reference Handler" - Thread t...@2
java.lang.Thread.State: WAITING on java.lang.ref.reference$l...@3b1799
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Locked ownable synchronizers:
- None
"main" - Thread t...@1
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked java.net.sockssocketi...@735aa1
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Locked ownable synchronizers:
- None
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
