Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
Hi I'm using Tomcat 5.0.26 that comes with JBoss 3.2.5 with default configurations A servlet creates pie/bar charts using JFreeChart. This is refreshed every 5 seconds via javascript refresh in the brower. All this has been workign fine for most customers. Suddenly for a specific customer sessions coming from a specific newtwork/georaphical location we see that the 5 seconds chart refreshes are hanging the thread. Charts are 20 KB. For a single user in the system sometimes these requests are talking up 20 ~30 threads and each stuck for 15~16 minutes before it finally clears up. We ran ethereal on the tcp packet and see that when the threads are hung, browser is sending FIN, ACK eventhough server hasn't completed sending the data. Thus we see TCP retranmissions of same packets as its not receiveing any ack from the client. How do I troubleshoot this issue. System is not memory starved and cpu is 98% idle. Any pointers on trouble shooting this would be greatly appreciated. Thanks Kumar Here's the thread dump of one of the hung request. java.net.SocketOutputStream.socketWrite0(Native java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) java.net.SocketOutputStream.write(SocketOutputStream.java:136) org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:714) org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398) org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:318) org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:737) org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:125) org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:569) org.apache.coyote.Response.doWrite(Response.java:542) org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:368) org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398) org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:318) org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:297) org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:85) sun.awt.image.codec.JPEGImageEncoderImpl.writeJPEGStream(Native locked sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:472) sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:228) locked sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:235) locked org.jfree.chart.encoders.SunJPEGEncoderAdapter.encode(SunJPEGEncoderAdapter.java:139) org.jfree.chart.encoders.EncoderUtil.writeBufferedImage(EncoderUtil.java:136) org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:442) org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:388) com.transerainc.rtmcui.util.ChartUtil.drawChartAsJPEG(ChartUtil.java:46) __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
Sorry I forgot to include couple of more stats Tomcat is running on linux FC4 Client browser is IE 6.0 First line of the dump is "http-0.0.0.0-80-Processor99" daemon prio=1 tid=0x8b09b4b8 nid=0x46ec runnable [0x84be7000..0x84be91c0] Thx --- Kumar Pandey <[EMAIL PROTECTED]> wrote: > Hi > I'm using Tomcat 5.0.26 that comes with JBoss > 3.2.5 > with default configurations > > A servlet creates pie/bar charts using JFreeChart. > This is refreshed every 5 seconds via javascript > refresh in the brower. All this has been workign > fine > for most customers. > Suddenly for a specific customer sessions coming > from > a specific newtwork/georaphical location we see that > the 5 seconds chart refreshes are hanging the > thread. > Charts are 20 KB. For a single user in the system > sometimes these requests are talking up 20 ~30 > threads > and each stuck for 15~16 minutes before it finally > clears up. > We ran ethereal on the tcp packet and see that when > the threads are hung, browser is sending FIN, ACK > eventhough server hasn't completed sending the data. > Thus we see TCP retranmissions of same packets as > its > not receiveing any ack from the client. > > How do I troubleshoot this issue. > System is not memory starved and cpu is 98% idle. > > Any pointers on trouble shooting this would be > greatly > appreciated. > > Thanks > Kumar > > Here's the thread dump of one of the hung request. > > java.net.SocketOutputStream.socketWrite0(Native > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > java.net.SocketOutputStream.write(SocketOutputStream.java:136) > org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:714) > org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398) > org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:318) > org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:737) > org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:125) > org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:569) > org.apache.coyote.Response.doWrite(Response.java:542) > org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:368) > org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398) > org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:318) > org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:297) > org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:85) > sun.awt.image.codec.JPEGImageEncoderImpl.writeJPEGStream(Native > locked > sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:472) > sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:228) > locked > sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:235) > locked > org.jfree.chart.encoders.SunJPEGEncoderAdapter.encode(SunJPEGEncoderAdapter.java:139) > org.jfree.chart.encoders.EncoderUtil.writeBufferedImage(EncoderUtil.java:136) > org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:442) > org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:388) > com.transerainc.rtmcui.util.ChartUtil.drawChartAsJPEG(ChartUtil.java:46) > > __ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam > protection around > http://mail.yahoo.com > > - > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
If it was synchronization issue at the code level then I should have seen the issue during load test in our env. We see the issue only for the traffic coming from this specific network even for just one user in the system. Cpu is 99% idle. Also IE just allows 2 socket connection from the browser at any given time for that instance of IE.I see upto 30 threads tied up to service one user session requests for 20KB of data. Each finally clears up after 16 minutes. Meanhile other users accessing from other networks (even from India) does not have this issue. Their request for same data is getting serviced under sub seconds. by remaining threads not tied up. Does anyone have any idea on the re-transmissions of the same packets from server tcp layer to broswer thus hanging the connection for upto 16 minutes in Linux FC4, Tomcat 5.0.28. Who is closing the connection after ~16 minutes etc. Thanks Kumar --- Rajeev Jha <[EMAIL PROTECTED]> wrote: > looking at http://www.jfree.org/jfreechart/javadoc/ > , > > org.jfree.chart.encoders.EncoderUtil.writeBufferedImage > does not look thread-safe. > try checking > sun.awt.image.codec.JPEGImageEncoderImpl.encode API. > > Try increasing the latency by re-requesting the > servlet every 30 secs/ 1 min and see if that works > fine or not. > > > > > Kumar Pandey wrote: > > > > >Sorry I forgot to include couple of more stats > > > >Tomcat is running on linux FC4 > >Client browser is IE 6.0 > > > >First line of the dump is > >"http-0.0.0.0-80-Processor99" daemon prio=1 > >tid=0x8b09b4b8 nid=0x46ec runnable > >[0x84be7000..0x84be91c0] > > > >Thx > > > > > > > > > - > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes -Fixed!!
OK we finally pinnned the issue down to a client's proxy server. We have few ajax refreshes happening on our html pages. Under certain conditions IE sends a RST when its unable to serve these requests (you can quickly verify this by running ethreal while viewing new Yahoo pages with real time updates on e-mail weather etc). These RST were not getting propageted by the proxy server to our server. Thus our server kept sending re-transmissions on that socket. Once the proxy was set to just do passthru , eveythign worked fine as the server received RST sent from the browser. Thanks Kumar Pandey --- Kumar Pandey <[EMAIL PROTECTED]> wrote: > If it was synchronization issue at the code level > then > I should have seen the issue during load test in our > env. > > We see the issue only for the traffic coming from > this > specific network even for just one user in the > system. > Cpu is 99% idle. > > Also IE just allows 2 socket connection from the > browser at any given time for that instance of IE.I > see upto 30 threads tied up to service one user > session requests for 20KB of data. Each finally > clears > up after 16 minutes. Meanhile other users accessing > from other networks (even from India) > does not have this issue. Their request for same > data > is getting serviced under sub seconds. by remaining > threads not tied up. > > Does anyone have any idea on the re-transmissions of > the same packets from server tcp layer to broswer > thus > hanging the connection for upto 16 minutes in Linux > FC4, Tomcat 5.0.28. Who is closing the connection > after ~16 minutes etc. > > Thanks > Kumar > > --- Rajeev Jha <[EMAIL PROTECTED]> wrote: > > > looking at > http://www.jfree.org/jfreechart/javadoc/ > > , > > > > > org.jfree.chart.encoders.EncoderUtil.writeBufferedImage > > does not look thread-safe. > > try checking > > sun.awt.image.codec.JPEGImageEncoderImpl.encode > API. > > > > Try increasing the latency by re-requesting the > > servlet every 30 secs/ 1 min and see if that works > > fine or not. > > > > > > > > > > Kumar Pandey wrote: > > > > > > > > >Sorry I forgot to include couple of more stats > > > > > >Tomcat is running on linux FC4 > > >Client browser is IE 6.0 > > > > > >First line of the dump is > > >"http-0.0.0.0-80-Processor99" daemon prio=1 > > >tid=0x8b09b4b8 nid=0x46ec runnable > > >[0x84be7000..0x84be91c0] > > > > > >Thx > > > > > > > > > > > > > > > > - > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > __ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam > protection around > http://mail.yahoo.com > > - > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
