During high load of 16k requests per minute, we notice below error in log. [error] 2437#2437: *13335389 upstream timed out (110: Connection timed out) while reading response header from upstream, server: jahez.net, request: "GET /serviceContext/ServiceName?callback= HTTP/1.1", upstream: " http://127.0.0.1:8080/serviceContext/ServiceName
Below is the flow of requests: cloudflare-->AWS ALB--> NGINX--> Tomcat-->Elastic-search In NGINX we have the below config location /serviceContext/ServiceName{ proxy_pass http://localhost:8080/serviceContext/ServiceName; proxy_http_version 1.1; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_buffers 16 16k; proxy_buffer_size 32k; } below is tomcat connector config <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol" connectionTimeout="200" maxThreads="50000" URIEncoding="UTF-8" redirectPort="8443" /> We monitor the open file using *watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l" *the number of tomcat open files keeps increasing slowing the responses. the only option to recover from this is to restart tomcat.