I'm having an interesting problem which seems to be Tomcat-related and
which occurs randomly under heavy load. I'd appreciate some advice/help.
I'm developing a Java web application and deploying to Tomcat 6.0.32. (I
get the same behaviour with the latest Tomcat 7).
The web app uses Spring 3.05 and has a RESTful interface implemented
with Jersey 1.8. It only runs on my machine and nothing else talks to it
apart from a Java integration test which throws data at it.
The app is a "black box" which is interested in counting combinations of
inputs from forms. Randomly, the counts in the database would be wrong
-- never ever less than expected, but always one or two more than
expected. In other words, it's not as if there is a race condition and
updates are being overwritten since the counts are never less than they
should be. The back-end is thoroughly thread-safe.
My test harness generates 10 batches of loads. Each batch is submitted
concurrently, and each request within a batch is submitted serially with
a 1/10 second delay. (The batch sizes are: 5372, 801, 3005, 637, 9311,
313, 1024, 777, 412, 1201.)
Anyway, I incorporated the following diagnostics in my application:
o various statistics using JMX beans and atomic integer counts
o atomic integer request count in the test harness
o http header in each request containing the unique request count
These diagnostics show that the test harness sends out the correct
number of requests but that the web app receives more requests than the
test harness is sending. I put a BitSet into the Jersey resource
endpoint (which receives each request), large enough for the total
number of requests, and set the appropriate bit corresponding to the
unique count I pass as an http header. Debugging shows that I sometimes
receive more than one request with the same count.
Why is Tomcat sending duplicate requests? Is there some timeout that is
silently happening under load and causing Tomcat to resend the request?
Are there any JMX diagnostics I can look at/turn on for Tomcat?
Something is sending those duplicate requests and it isn't me!
cheers,
Peter
