Ayub,
On 10/28/20 23:28, Ayub Khan wrote:
During high load of 16k requests per minute, we notice below error in log.
[error] 2437#2437: *13335389 upstream timed out (110: Connection timed
out) while reading response header from upstream, server: jahez.net,
request: "GET /serviceContext/ServiceName?callback= HTTP/1.1", upstream: "
http://127.0.0.1:8080/serviceContext/ServiceName
Below is the flow of requests:
cloudflare-->AWS ALB--> NGINX--> Tomcat-->Elastic-search
I'm curious about why you are using all of cloudflare and ALB and nginx.
Seems like any one of those could provide what you are getting from all
3 of them.
In NGINX we have the below config
location /serviceContext/ServiceName{
proxy_pass http://localhost:8080/serviceContext/ServiceName;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffers 16 16k;
proxy_buffer_size 32k;
}
What is the maximum number of simultaneous requests that one nginx
instance will accept? What is the maximum number of simultaneous proxied
requests one nginx instance will make to a back-end Tomcat node? How
many nginx nodes do you have? How many Tomcat nodes?
below is tomcat connector config
<Connector port="8080"
protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="200" maxThreads="50000"
URIEncoding="UTF-8"
redirectPort="8443" />
50,000 threads is a LOT of threads.
We monitor the open file using *watch "sudo ls /proc/`cat
/var/run/tomcat8.pid`/fd/ | wc -l" *the number of tomcat open files keeps
increasing slowing the responses. the only option to recover from this is
to restart tomcat.
So this looks like Linux (/proc filesystem). Linux kernels have a 16-bit
pid space which means a theoretical max pid of 65535. In practice, the
max pid is actually to be found here:
$ cat /proc/sys/kernel/pid_max
32768
(on my Debian Linux system, 4.9.0-era kernel)
Each thread takes a pid. 50k threads means more than the maximum allowed
on the OS. So you will eventually hit some kind of serious problem with
that many threads.
How many fds do you get in the process before Tomcat grinds to a halt?
What does the CPU usage look like? The process I/O? Disk usage? What
does a thread dump look like (if you have the disk space to dump it!)?
Why do you need that many threads?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
