Chris, Nginx is being used for image caching and converting https to http requests before hitting tomcat.
The behavior I am noticing is application first throws Borken pipe client abort exception at random apis calls followed by socket timeout and then database connection leak errors. This happens only during high load. During normal traffic open files for tomcat process goes up and down to not more than 500. However during high traffic if I keep track of the open files for each tomcat process as soon as the open files count reaches above 10k that tomcat instance stops serving the requests. If the open file count goes beyond 5k its sure that this number will never come back to below 500 at this point we need to restart tomcat. There are three application installed on each tomcat instance, 1) portal: portal calls (2) and (3) using localhost, should we change this to use dns name instead of localhost calls ? 2) Services for portal 3) Services for portal and mobile clients On Thu, May 28, 2020 at 4:50 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Ayub, > > On 5/27/20 19:43, Ayub Khan wrote: > > If we have 18 core CPU and 100GB RAM. What value can I set for > > maxConnections ? > Your CPU and RAM really have nothing to do with it. It's more about > your usage profile. > > For example, if you are serving small static files, you can serve a > million requests a minute on a Raspberry Pi, many of them concurrently. > > But if you are performing fluid dynamic simulations with each request, > you will obviously need more horsepower to service a single request, > let alone thousands of concurrent requests. > > If you have tons of CPU and memory to spare, feel free to crank-up the > max connections. The default is 10000 which is fairly high. At some > point, you will run out of connection allocation space in the OS's > TCP/IP stack, so that is really your upper-limit. You simply cannot > have more than the OS will allow. See > https://stackoverflow.com/a/2332756/276232 for some information about > that. > > Once you adjust your settings, perform a load-test. You may find that > adding more resources actually slows things down. > > > Want to make sure we are utilizing the hardware to the max > > capacity. Is there any config of tomcat which enabled could help > > serve more requests per tomcat instance. > > Not really. Improving performance usually come down to tuning the > application to make the requests take less time to process. Tomcat is > rarely the source of performance problems (but /sometimes/ is, and > it's usually a bug that can be fixed). > > You can improve throughput somewhat by pipelineing requests. That > means HTTP keepalive for direct connections (but with a small timeout; > you don't want clients who aren't making any follow-up requests to > waste your resources waiting for a keep-alive timeout to close a > connection). For proxy connections (e.g. from nginx), you'll want > those connections to remain open as long as possible to avoid the > re-negotiation of TCP and possibly TLS handshakes. Using HTTP/2 can be > helpful for performance, at the cost of some CPU on the back-end to > perform the complicated connection management that h2 requires. > > Eliminating useless buffering is often very helpful. That's why I > asked about nginx. What are you using it for, other than as a barrier > between the load-balancer and your Tomcat instances? If you remove > nginx, I suspect you'll see a measurable performance increase. This > isn't a knock against nginx: you'd see a performance improvement by > removing *any* reverse-proxy that isn't providing any value. But you > haven't said anything about why it's there in the first place, so I > don't know if it /is/ providing any value to you. > > > The current setup is able to handle most of the load, however > > there are predictable times where there is an avalanche of requests > > and thinking how to handle it gracefully. > > You are using AWS: use auto-scaling. That's what it's for. > > - -chris > > > On Wed, May 27, 2020 at 5:38 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Ayub, > > > > On 5/27/20 09:26, Ayub Khan wrote: > >>>> previously I was using HTTP/1.1 connector, recently I > >>>> changed to NIO2 to see the performance. I read that NIO2 is > >>>> non blocking so trying to check how this works. > > > > Both NIO and NIO2 are non-blocking. They use different strategies > > for certain things. Anything but the "BIO" connector will be > > non-blocking for most operations. The default is NIO. > > > >>>> which connector protocol do you recommend and best > >>>> configuration for the connector ? > > This depends on your environment, usage profile, etc. Note that > > non-blocking IO means more CPU usage: there is a trade-off. > > > >>>> Which stable version of tomcat would you recommend ? > > > > Always the latest, of course. Tomcat 8.0 is unsupported, replaced > > by Tomcat 8.5. Tomcat 9.0 is stable and probably the best version > > if you are looking to upgrade. Both Tomcat 8.5 and 9.0 are > > continuing to get regular updates. But definitely move away from > > 8.0. > > > >>>> Are there any ubuntu specific configs for tomcat ? > > No. There is nothing particular special about Ubuntu. Linux is one > > of the most well-performing platforms for the JVM. I wouldn't > > recommend switching platforms. > > > > Why are you using nginx? You already have load-balancing happening > > in the ALB. Inserting another layer of proxying is probably just > > adding another buffer to the mix. I'd remove nginx if it's not > > providing any specific, measurable benefit. > > > >>>> We are using OkHttp client library to call rest api and > >>>> stack trace shows failure at the api call. The api being > >>>> called is running on the same tomcat instance (different > >>>> context) usring url localhost. This does not happen when > >>>> number of requests is less. > > > > Your Tomcat server is calling this REST API? Or your server is > > serving those API requests? If your service is calling itself, > > then you have to make sure you have double-capacity: every > > incoming request will cause a loopback request to your own > > service. > > > > Other than the timeouts, are you able to handle the load with your > > existing infrastructure? Sometimes, the solution is simply to > > throw most hardware at the problem. > > > > -chris > > > >>>> On Wed, May 27, 2020 at 11:48 AM Mark Thomas > >>>> <ma...@apache.org> wrote: > >>>> > >>>>> On 26/05/2020 23:28, Ayub Khan wrote: > >>>>>> Hi, > >>>>>> > >>>>>> During high load I am seeing below error on tomcat logs > >>>>>> > >>>>>> java.util.concurrent.ExecutionException: java.net > >>>>> .SocketTimeoutException: > >>>>>> timeout > >>>>> > >>>>> And the rest of that stack trace? It is hard to provide > >>>>> advice without context. We need to know what is timing out > >>>>> when trying to do what. > >>>>> > >>>>>> We have 4 C5.18x large vms running tomcat 8 behind AWS > >>>>>> application load balancer. We are seeing socket timeouts > >>>>>> during peak hours. What should be the configuration of > >>>>>> tomcat if we get 60,000 to 70,000 requests per > >>>>> minute > >>>>>> on an average ? > >>>>>> > >>>>>> Tomcat 8.0.32 on Ubuntu 16.04.5 LTS > >>>>> > >>>>> Tomcat 8.0.x is no longer supported. > >>>>> > >>>>>> Below is the java version: > >>>>>> > >>>>>> java version "1.8.0_181" Java(TM) SE Runtime Environment > >>>>>> (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM > >>>>>> (build 25.181-b13, mixed mode) > >>>>>> > >>>>>> > >>>>>> Below is the server.xml connector configuration: > >>>>>> > >>>>>> <Connector port="8080" > >>>>>> protocol="org.apache.coyote.http11.Http11Nio2Protocol" > >>>>> > >>>>> Why NIO2? > >>>>> > >>>>> Mark > >>>>> > >>>>> > >>>>>> connectionTimeout="20000" > >>>>>> > >>>>>> URIEncoding="UTF-8" redirectPort="8443" /> > >>>>>> > >>>>>> We have 4 C5.18x large vms and each vm has nginx and > >>>>>> tomcat instance running. All the 4 vms are connected to > >>>>>> AWS application load balancer. > >>>>>> > >>>>>> I tried to add maxConnections=50000 but this does not > >>>>>> seem to have any affect and still saw the timeouts > >>>>>> > >>>>>> Thanks and Regards Ayub > >>>>>> > >>>>> > >>>>> > >>>>> ------------------------------------------------------------------ > - --- > >>>>> > >>>>> > > > >>>>> > >>>>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>> For additional commands, e-mail: > >>>>> users-h...@tomcat.apache.org > >>>>> > >>>>> > >>>> > >> > >> --------------------------------------------------------------------- > >> > >> > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7PwX0ACgkQHPApP6U8 > pFhX9RAAyfSvUt/rU4xist6W2c8wcvC6dg7qQ1WhQ0yl68Jm7PNkVTffpNQzIh6t > vD6TUvIckF1O0AeWIwRXpfgFPUc0DUBiyrkEPwnQsiNMjkhMtq4iuESiFb5i/2NK > 0J3JUHYusIwjycmfXeRrkVIG9ei9339bklclzHc3+DZpgERF7FQ+JvhloDb+uv9h > NOmKwxXHri7Ikcqt8vSbkPadYsmjTPnj+eGjcS5mYSi9/W2OoVLxz1NB32iUoIi0 > 3Y0L78DvddbPYBaJ7WzkxiqKcTBfhBhFeV2PJKY8WqLUQXJsDc5SIVESh++TIeCx > gD0OPjnP8XTgyUp4BTJnP604SAI+egIRzmLDKJYlv+KKb6y1GqQgPUuiMcLwFmb5 > 7T+h6R805bZJAdnp8ToM+g8FJR4/cZcp8uwR7zLPeUYq8KNPPWG8pUGk7byg7h8J > 8gRCQ75j5+jqjKtweEyncBi0rKS6QN73SKh1crCVJOy/1QymsD/9COyChfFximPh > rAb9S35rcxzGq+sEF6d/TcIvQ/Zso6fVA29QXgKBUwW1QS8GQu9CY/odYDchBlVA > Kv1KsDPWMFPPraVvnckDk8KKs9dwRCtIn9QzX2MgOGhIFK4TJbs7+PsAnchHWMLh > dhyPx+HDygCKvx33UyXtZNEitqxx1FVJa+W6zPxwKCOT7ZQS2gQ= > =PIuS > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --
