> Paul, Thanks for the reply. I am not really much further with my main issue, but I hope this reply provides more information to you, so you can either clear up my confusion or see how tomcat doesn't work as intended in my case.
> On 5/18/21 07:44, Paul P Wolf wrote: > > Hi, > > > > I am trying to run a spring boot application with an embedded tomcat. In a > scenario, where there is a lot of load on the service, I would like tomcat to > refuse connections or return something like a 503 Service Unavailable. From > what I understand, you could have this behaviour by setting maxConnections > and and any additional connections get refused. At least this was how the old > blocking io acceptor worked, from what I understand. > > > > The documentation says "The maximum number of connections that the > server will accept and process at any given time. When this number has been > reached, the server will accept, but not process, one further connection. This > additional connection be blocked until [...]". > > However the documentation doesn't really state what happens if > maxConnection+2 connections are reached. > > > > I tried to run my application with following settings (embedded-tomcat > 9.0.45): > > maxConnections=3 > > acceptCount=2 > > maxThreads=1 > > processorCache=1 > > > > I created an endpoint, which just sleeps for a few seconds and returns > > a string. When I create 50 separate connection via curl instances to > > call that service I see the following behaviour with the NIO Acceptor: > > > > * 6 http connections are accepted immediately (maxThreads + > > acceptCount + maxConnections) > The maxThreads setting should not be relevant, here. maxConnections > counts the total connections without regard to how many of them are > actually having work done (which requires a thread from maxThreads). > > So accepting 6 connections really means: > > maxConnections (3) + acceptCount (2) + 1 (because maxConnections says it > will accept 1+maxConnections, which is a little confusing IMO). > I tried some different settings to check this and your interpretation seems to be correct. Thanks! > > * 44 http connections aren't established just, but neither are they > > refused. I will call them "blocked", but different from the > > specification those are 44 blocked connections and not just 1 > What is the TCP state of the first 6 connections? What about the other 44? > What is the difference between "accepted" and "blocked", and how are you > telling them apart? I tell them apart by inferring from curl's behaviour. I have a service endpoint sleep for 10s. I start 50 curls in parallel with a 5s connect-timeout. 6 connections go through (after a total of 60 seconds), while the other 44 fail after the 5s timeout. Makes sense? By that I infer that 44 connections are in the SYN-SENT state and the other 6 are in the ESTABLISHED state, after the curl instances have started. > > * once the first request finishes, the latest (blocked) requests > > gets a connection and is being processed (not a request from the > > accept-queue or one of the other already established connections) > So the queue is behaving "unfairly"? > > > * when there are no further blocked requests, the requests still > > get processed in last in first out order > More unfairness. > > You didn't post your <Connector> configuration (which is pretty critical for > trying to think about these things), but I suspect you aren't using an > <Executor>, which may ensure fairness. (Older Tomcats used an internal > thread pool which was NOT an "Executor" but later Tomcat should always be > using an Executor, which was intended to enforce fairness. Hmm.) Correct, it behaves unfairly. As stated I am using spring boot and an embedded tomcat. From why I understand there is no (obvious) way to provide traditional tomcat configuration files. However I used an TomcatConnectorCustomizer to inspect the connector and there seems to be no Executor configured. Would you say then, that there is a problem with the default spring boot configuration not using an Executor? > > * I see some timeouts after a while with this setup, depending on > > what timeouts I set on curl. The requests without an established > > connection timeout with "connect-timeout" parameter and the ones with > > established connections depending on the "max-time" parameter. > When you get a timeout, what kind of timeout are you encountering (on the > client side)? Is this a "connect timeout" or a "read timeout"? I encounter both, depending on the configuration. My point here was that I could use those timeouts to infer the tcp state as stated above. > > Now I have a lot of questions and I suspect some bugs, but I wanted to ask > on this list first: > > > > 1. Is there a way to refuse connections, instead of blocking them? > > maxConnections + acceptCount *should* be doing this. Remember that > acceptCount is just a suggestion to the OS. The OS is free to always accept > 65535 connections into its TCP/IP stack whether you say so or not. Likewise, > if > you are connecting through other network devices (e.g. > load-balancer, whatever), there may be different rules in different places. > I'm assuming you are connecting directly to Tomcat on localhost, but you > didn't specify. You are correct in assuming that I tested everything on localhost. So do I understand you correctly? On one hand tomcat would behave as intended by me, but on the other hand, I can't really configure the acceptCount? How would I figure out what the actual accept count of my OS is? If acceptCount turns out to be greater than 16 or so that would be bad for my situation. However I don't really think, that the OS overrides the acceptCount, because I can see the effect of changing the parameter by observing the number of ESTABLISHED connections. So this rather seems like a bug to me considering those assumptions. > > 2. I can't control the connect-timeout of the clients - is there a > > way to control it on the server side? (I tried to change > > keepAliveTimeout and connectionTimeout without success) > It's impossible for the server to instruct the client to time-out. The > situation > here is that the client can't reach the server therefore the server can't > communicate anything to the client. > > If you can't change the connect-timeout of the clients, do you know what the > effective timeout actually is? Only in my local test setup > > 3. Why doesn't the specification state what happens if many more > > requests are being made than maxConnections allows for? (esp. with the > > behaviour being different among acceptors) > Which specification? The Tomcat documentation explains what happens with > 0-maxConnections, then maxConnections + 1, then maxConnection + N > where N > 1, so ... what are you actually asking? I am refering to http://tomcat.apache.org/tomcat-9.0-doc/config/http.html. I didn't find an explanation before, but now I see "[...] Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them.". I will try to summarize my understanding and problems with the configuration parameters bellow > > 4. The processing order is troublesome - early requests get starved > > by following ones. Why is this lifo instead of fifo? > Please try explicitly configuring <Executor> and see if that changes anything. as explained above, there is not <Executor> config in spring boot, but it seems no executor is set by default!? > > 5. The number of accepted connections is unintuitively. Why are > > active connections with requests either being processed or being in > > the accept queue not counted as active connections (actual connections > > = maxConnections + acceptCount + maxThreads)? > Tomcat cannot see the OS's TCP backlog (the "accept queue") so nothing can > be counted there. The only thing Tomcat can see is up to maxConnections + 1 > at which point the acceptor thread will stop accepting incoming connections > (and those client connections will queue-up in the TCP stack's backlog). > > > 6. What is the difference between requests in the accept queue and > > requests outside the queue, but still within the maxConnections limit? > Let's be clear about nomenclature. When you say "accept queue", I think > "TCP stack backlog" which is 100% in the OS and not visible to Tomcat. > Tomcat itself has its own queue of incoming connections (ones that have > been accept()ed but no thread has yet been assigned). I suppose you mean > this Tomcat-queue when you say "requests outside the queue, but still > within the maxConnections limit"? > > The difference is that the former (backlog) is not visible to Tomcat at all > (or > any server that you might be using, this is not Tomcat- or > Java-specific) and the latter (having been accept()ed by Tomcat) are being > managed by Tomcat. In regards to point 5 and 6, let me try to point out my issues with the documentation and your explanations: - "Each incoming request requires a thread for the duration of that request. If more simultaneous requests are received than can be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute)." So far the documentation sounds good to me. - "If still more simultaneous requests are received, they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute)." This is what I meant with "accept queue" or what you call "Tomcat's own queue". Would you usually configure the acceptCount to be less than maxConnections or is it completely unrelated to maxConnections? I would intuitively assume you would set it to less, but now I am not sure anymore. - "Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them." Now if that is the case, than what does maxConnection have to do with anything at all? Also I just don't see any connections being refused, but instead they linger in SYN-SENT state. You say, that acceptCount can be overridden by the OS, but why would that take away tomcats possibility of monitoring those connections, as "they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute)"? The way I see it, tomcat should always be able to monitor those connections or never at. Are there separate acceptCounts for Tomcat/OS/TCP stack? - "Note that once the [maxConnection] limit has been reached, the operating system may still accept connections based on the acceptCount setting". Here again my confusion rises: Does this only apply/make sense if the OS overrides the acceptCount? If so, would Tomcat still be able to monitor those connections in the server socket created by the Connector or not? If the OS doesn't override the acceptCount, is Tomcat then able to monitor the connections? Furthermore: The documentation implies that if acceptCount < maxConnections, than it will only allow maxConnections, but as you pointed out yourself earlier, this doesn't seem to be the case, but instead it's "maxConnections (3) + acceptCount (2) + 1". What suggests to me that the OS doesn't override the acceptCount in my case is, that the ESTABLISHED connections always correlate to my acceptCount setting, but still connections don't get refused. This seems like a bug to me. - "When you say "accept queue", I think "TCP stack backlog" which is 100% in the OS and not visible to Tomcat. Tomcat itself has its own queue of incoming connections (ones that have been accept()ed but no thread has yet been assigned). I suppose you mean this Tomcat-queue when you say "requests outside the queue, but still within the maxConnections limit"?"" - with accept queue I mean the internal one. How do you know what goes into the TCP stack backlog and what goes into the internal queue? What I mean by "requests outside the queue, but still within the maxConnections limit" is the following: Imagine your acceptCount is 5, but you accept more, because maxConnection 13. So maxConnection - acceptCount = 8. What happens with those requests? Do they land in the TCP stack backlog? As discussed before, I actually see 19 ESTABLISHED connections with a configuration like that. So do actually 13 land in the backlog? To me this seems like a (documentation) bug. I hope this makes it clearer. > > > Any insight would be appreciated. If there is anything in here, that I > > should report as bug, please let me know. I am happy to file one. > > We will need more information. In your next post, please provide either your > <Connector> if you are using XML-based configuration or all your code that is > creating/configuring the Tomcat Connector. Also, please give some default > about your JVM, OS, and kernel version, etc. I don't use XML config and use the default settings of embedded tomcat 9.0.45 as shipped with spring-boot 2.4.5 The JVM is: openjdk 11.0.9.1 The kernel version is: 5.4.0-73-generic If any additional info is needed, let me know. ________________________________ Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20210430> Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz
