Re: AW: AW: AW: maxConnections behaving unexpected - no connection gets ever refused

[Christopher Schultz]

Paul,

On 5/19/21 08:32, Paul P Wolf wrote:
sorry, my message was misformated, so here again with hopefully better 
formatting:
The clients timeout because they spend more than timeout in the
acceptCount/backlog queue waiting for Tomcat to call Socket.accept()

Ok, so you are stating, that clients timeout while their requests are in the 
acceptCount/backlog. This is not what I am seeing. See below.

So we have:
maxThreads=4
maxConnections=10
acceptCount=20

and a request processing time of 1 second.

I'd guess that the OS is using a much large accept count. Let's model it.

0s - 50 connections in acceptCount
1s - 39 connections in acceptCount, 11 connections maintained by Tomcat,
       4 requests processing
2s - 35 connections in acceptCount, 11 connections maintained by Tomcat,
       4 requests processing, 4 completed requests 3s - 31 connections
in acceptCount, 11 connections maintained by Tomcat,
       4 requests processing, 8 completed requests 4s - 27 connections
in acceptCount, 11 connections maintained by Tomcat,
       4 requests processing, 12 completed requests 5s - 23 connections
in acceptCount, 11 connections maintained by Tomcat,
       4 requests processing, 16 completed requests 6s - 19 connections
timed out, 11 connections maintained by Tomcat,
       4 requests processing, 20 completed requests 7s - 19 connections
timed out, 7 connections maintained by Tomcat,
       4 requests processing, 24 completed requests 8s - 19 connections
timed out, 3 connections maintained by Tomcat,
       3 requests processing, 28 completed requests 9s - 19 connections
timed out, 0 connections maintained by Tomcat,
       0 requests processing, 31 completed requests

That seems to match what you observed. That suggests the OS is using
an acceptCount of at least 50.

I can see what you mean, but this is not what I see, because your assumption of 
a 1s processing time is wrong. The processing time of each request is 10s 
(thanks to a 10s sleep, which blocks the Thread).

So here is what I see instead (note I don't guess the response time, but do 
actually see/measure it):

0s - 4 requests processing, 11 connections maintained, 20 connections in 
acceptCount, 19 not in acceptCount

5s - 4 requests processing, 11 connections maintained, 20 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out

10s - 4 requests processing, 11 connections maintained, 16 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 4 requests processed

20s - 4 requests processing, 11 connections maintained, 12 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 8 requests processed

30s - 4 requests processing, 11 connections maintained, 8 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 12 requests processed

40s - 4 requests processing, 11 connections maintained, 4 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 16 requests processed

50s - 4 requests processing, 11 connections maintained, 0 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 20 requests processed

60s - 4 requests processing, 7 connections maintained, 0 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 24 requests processed

70s - 3 requests processing, 3 connections maintained, 0 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 28 requests processed

80s - 0 requests processing, 0 connections maintained, 0 connections in 
acceptCount, 0 not in acceptCount, 19 timed-out, 31 requests processed

The interesting thing to note is, that contrary to your statement, I don't see 
timeouts for requests in the acceptCount/backlog.

To verify my interpretation, I increased the acceptCount and I saw that the 
time outs decrease by the same number. I decreased the acceptCount and saw that 
the time outs increase.

The behaviour seems 100% predictable to me... just not as expected based on the 
documentation.

Not up to Tomcat. Tomcat can only call Socket.accept() and does so
under the control of maxConnections.

Connection refused == acceptCount/backlog full (or no listening socket).

Connection refusal is entirely under the control of the OS and will be
driven largely by the actual value of acceptCount/backlog.
So if it is not up to Tomcat, then the documentation "Any further simultaneous requests will 
receive "connection refused" errors" is clearly wrong, isn't it?

I realize we've moved-on a bit from here, but I wanted to make sure we 
were all using the same language about timeouts and such. Showing your 
curl command-line would be helpful. Under certain situations, curl will 
retry connections, etc. and knowing exactly what you are handing to curl 
would be helpful.

I would certainly expect that if you have maxConnections=10 and 
acceptCount=10 and your service just sleeps for 10 seconds and you have 
a 5-second client timeout, then:

* 11 connections are immediately accepted by Tomcat, allocated a thread 
and immediately start sleeping (ESTABLISHED)
* 10 connections are immediately put into the TCP stack's backlog 
(ESTABLISHED, to my surprise!)
* 9 connections are refused (connect error)

After 5 seconds:
* All 21 remaining clients time-out

If the above is not what is happening, then I would suspect:

1. The OS is using a bigger backlog than requested
2. curl is retrying
3. We are misunderstanding how curl's settings affect connect/read timeouts

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org