> > 1) If you want to analyze your original problem, you need to get back to > the original situation, i.e. without connectionTimeout. It doesn't make > much sense to guess about the original problem by looking at something > very different.
[Pantvaidya, Vishwajit] Yes - I have already initiated that. Expect to be able to test the system without connectionTimeout tonight. > 2) The output of netstat and the content of a thread dump change in > time. If you want to understand the exact relations between the two > netstat outputs and a thread dump, you need to ensure to produce those > three things as close in time as possible. I'm talking about seconds not > milliseconds. [Pantvaidya, Vishwajit] Yes, I had taken the netstats and the threads dumps pretty close together. But I will try and append a timestamp next time I take the output. > 3) I think I already indicated that you do not want to look at entries > in TIME_WAIT state. This state is special and not related to any threads [Pantvaidya, Vishwajit] My netstat o/p had FIN_WAIT and CLOSE_WAIT, but not TIMED_WAIT. Did some reading on TCP states, and seems to me that next time I do a netstat (w/o the connTimeout in server.xml), I can ignore processes with all these wait states as they seem to just indicate connections in different stages of closing. > 4) Firewall idle connection drop: First read > > http://tomcat.apache.org/connectors- > doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping > > carefully and try to understand. > > Any mod_jk attribute that takes a booelan value will accept 1, true, > True, t or T as true, 0, false, False, f or F as false (and maybe even > more). [Pantvaidya, Vishwajit] Our workers.props file has most recommended settings: Worker...type=ajp13 Worker...cachesize=10 Worker...cache_timeout=600 Worker...socket_keepalive=1 Worker...recycle_timeout=300 We are not setting connectionpoolsize and minsize - but from the timeouts doc that should be okay as JK auto adjusts to httpd poolsize. So I think only thing left is to remove connectionTimeout and test. Your link recommends connectionTimeouts and JKoption +DisableReuse as a final option - I think that will remove persistent connections on httpd and tomcat side. For us, the connectionTimeout alone worked. And my netstat o/p showing 11 conns on http side and only 2 on tomcat side means our http conn's are persistent while tomcat ones are not, right?. So I am thinking the perf downside should be better than if I had set +DisableReuse also? > 5) Matching port numbers > > Usually the port numbers should match. The non matching of the port > numbers could indicate, that there is a firewall in between, although > most firewall systems will be transparent to the ports (yes, I know > there are many variations). Since the port numbers are very close I > would guess, that the reason for not matching is that netstat was done a > couple of seconds or more apart, and your connections are only used for > a very short time, so we have lots of new connections activity. [Pantvaidya, Vishwajit] Yes - I confirmed from admins that there is a firewall and I will work with them to understand that side more. Our connection timeout are in the order of 10 mins - so I am not sure why the port#s don't match - will try and find if the firewall is having different port# ranges configured for httpd & tomcat side. > 6) TCP states > > LISTEN on the Tomcat side corresponds to the one TP thread, that does a > socket accept. > > ESTABLISHED: both sides still want to use this connection. On the Tomcat > side shows up as socketRead0() > > CLOSE_WAIT: the other side has closed the connection, the local side > hasn't yet. E.g. if Tomcat closes the connection because of > connectionTimeout, but Apache doesn't have a configured idle timeout and > didn't yet try to reuse the half-closed connection, the connection will > be shown as CLOSE_WAIT on the httpd side. If Apache closed the > connection, but Tomcat hasn't noticed yet, it will be CLOSE_WAIT at the > Tomcat end. In this case it could be also socketRead0() in the thread > dump. > > FIN_WAIT2: most likely the other end of CLOSE_WAIT. [Pantvaidya, Vishwajit] This is interesting because all the conn's in the netstat o/p on httpd and tomcat sides which involved the connector port 21065 (either in local/foreign addr) were in WAIT states. But I was seeing one RUNNABLE thread in socketAccept in the thread console. But anyway I will redo this whole thing with the conntimeouts removed and make sure I take the netstats and thread dumps in close succession. > 7) mod_jk update > > Before you start to fix your mod_jk configuration, go to your ops people > and tell them that they are using a very bad mod_jk version and they > have to update. The right version to update to is 1.2.28. It does make > no sense at all to try to fix this with your old version. Solve your > problem in the right way, by setting much more attributes on the JK side > than simply the connectionTimeout on the Tomcat side. [Pantvaidya, Vishwajit] Will convey this again. > Most important: read the above timeouts page fully. [Pantvaidya, Vishwajit] Had done that earlier - but will do it again. Thanks a lot. Vish. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
