-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Ayub,
On 6/23/20 19:17, Ayub Khan wrote:
> Yes we have nginx as reverse proxy, below is the nginx config. We
> notice this issue only when there is high number of requests,
> during non peak hours we do not see this issue.> location
> /myapp/myservice{ #local machine proxy_pass
> http://localhost:8080; proxy_http_version 1.1;
>
> proxy_set_header Connection $connection_upgrade;
> proxy_set_header Upgrade $http_upgrade;
> proxy_set_header Host $host; proxy_set_header
> X-Real-IP $remote_addr; proxy_set_header
> X-Forwarded-For $proxy_add_x_forwarded_for;
>
>
> proxy_buffers 16 16k; proxy_buffer_size 32k; }
You might want to read about tuning nginx to drop connections after a
certain period of time, number of requests, etc. Looks like either a
bug in nginx or a misconfiguration which allows connections to
stick-around like this. You may have to ask the nginx people. I have
no experience with nginx myself, while others here may have some
experience.
> location / { # if using AWS Load balancer, this bit checks for the
> presence of the https proto flag. if regular http is found, then
> issue a redirect
to hit
> the https endpoint instead if ($http_x_forwarded_proto != 'https')
> { rewrite ^ https://$host$request_uri? permanent; }
>
> proxy_pass http://127.0.0.1:8080; proxy_http_version
> 1.1;
>
> proxy_set_header Connection $connection_upgrade;
> proxy_set_header Upgrade $http_upgrade;
> proxy_set_header Host $host; proxy_set_header
> X-Real-IP $remote_addr; proxy_set_header
> X-Forwarded-For $proxy_add_x_forwarded_for;
>
>
> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>
> *below is the connector*
>
> <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="2000" maxThreads="50000" URIEncoding="UTF-8"
> redirectPort="8443" />
50k threads is a LOT of threads. Do you expect to handle 50k requests
simultaneously?
> these ports are random, I am not sure who owns the process.
>
> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port 55866
> is a random port.
I'm sure you'll find that 55866 is owned by nginx. netstat will tell you
.
I think you need to look at your nginx configuration. It would also be
a great time to upgrade to a supported version of Tomcat. I would
recommend 8.5.56 or 9.0.36.
- -chris
> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Ayub,
>
> On 6/23/20 16:23, Ayub Khan wrote:
>>>> I executed *sudo lsof -p $(cat /var/run/tomcat8.pid) *and I
>>>> saw the below output, some in CLOSE_WAIT and others in
>>>> ESTABLISHED. If there are 200 open file descriptors 160 are
>>>> in CLOSE_WAIT state. When the count for CLOSE_WAIT increases
>>>> I just have to restart tomcat.
>>>>
>>>> java 65189 tomcat8 715u IPv6 237878311
>>>> 0t0 TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java
>>>> 65189 tomcat8 716u IPv6 237848923 0t0
>>>> TCP localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>
> These are connections from some process into Tomcat listening on
> port 8080 (that's what localhost:http-alt is). So what process owns
> the outgoing connection on port 40568 on the same host?
>
> Are you using a reverse proxy?
>
>>>> most of the open files are in CLOSE_WAIT state I do not see
>>>> anything related to database ip.
>
> Agreed. It looks like you have a reverse proxy who is losing-track
> of connections, or who is (re)opening connections when it may be
> unnecessar y.
>
> Can you share your <Connector> configuration from server.xml?
> Remember to remove any secrets.
>
> -chris
>
>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
>>>> felix.schumac...@internetallee.de> wrote:
>>>>
>>>>>
>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
>>>>>> Felix,
>>>>>>
>>>>>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>>>>> and from the
>>>>> output
>>>>>> I see majority of them are related to sockets as shown
>>>>>> below, some of
>>>>> them
>>>>>> point to the jar file of tomcat and others to the log
>>>>>> file which is
>>>>> created.
>>>>>>
>>>>>> socket:[2084570754] socket:[2084579487]
>>>>>> socket:[2084578478] socket:[2084570167]
>>>>>
>>>>> Can you try the other command (lsof -p $(cat
>>>>> ...tomcat.pid))? It should give a bit more details on the
>>>>> used sockets that the proc directory.
>>>>>
>>>>> Felix
>>>>>
>>>>>>
>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
>>>>>> felix.schumac...@internetallee.de> wrote:
>>>>>>
>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>>>>>>>> Chris,
>>>>>>>>
>>>>>>>> I am using HikariCP for connection pooling. If the
>>>>>>>> database is leaking connections then I should see
>>>>>>>> connection not available exception.
>>>>>>>>
>>>>>>>> How do I find out which file descriptors are leaking
>>>>>>>> ? these are not
>>>>>>> files
>>>>>>>> open on disk as there is no explicit disk file I/O in
>>>>>>>> this application.
>>>>>>>>
>>>>>>>> I just use the below command to check for open file
>>>>>>>> descriptors:
>>>>>>>>
>>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ |
>>>>>>>> wc -l"
>>>>>>> You could have a look at the name of the files in the
>>>>>>> pids proc
>>>>> directory.
>>>>>>>
>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>>>>>>
>>>>>>> Or you could use the tool lsof to find the open file
>>>>>>> descriptors.
>>>>>>>
>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
>>>>>>>
>>>>>>> For both calls you should first change to the uid of
>>>>>>> the tomcat user or use sudo as in your example.
>>>>>>>
>>>>>>> Felix
>>>>>>>
>>>>>>>> Thanks and Regards Ayub
>>>>>>>>
>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz
>>>>>>>> < ch...@christopherschultz.net> wrote:
>>>>>>>>
>>>>>>>> Ayub,
>>>>>>>>
>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
>>>>>>>>>>> Sorry we are using 8.0.32 version of tomcat.
>>>>>>>>>>>
>>>>>>>>>>> below is the configuration:
>>>>>>>>>>>
>>>>>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu)
>>>>>>>>>>> Server built: Jan 24 2020 16:24:30 UTC Server
>>>>>>>>>>> number: 8.0.32.0 OS Name: Linux OS Version:
>>>>>>>>>>> 4.4.0-1087-aws Architecture: amd64 JVM
>>>>>>>>>>> Version: 1.8.0_181-b13 JVM Vendor: Oracle
>>>>>>>>>>> Corporation
>>>>>>>>>>>
>>>>>>>>>>> I use the below command to check the file
>>>>>>>>>>> descriptors:
>>>>>>>>>>>
>>>>>>>>>>> watch "sudo ls /proc/`cat
>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>>>>>>>> So you know there is some kind of increase in
>>>>>>>> file-handle use, but you don't know what types of
>>>>>>>> file handles are increasing, right?
>>>>>>>>
>>>>>>>> Can you try to find out which kinds of file handles
>>>>>>>> are increasing?
>>>>>>>>
>>>>>>>> I have a sneaking suspicion that it's your database
>>>>>>>> connections and not actually files open on the disk.
>>>>>>>>
>>>>>>>> Are you using a database connection pool? If not,
>>>>>>>> you should really use one and limit the number of
>>>>>>>> connections to something sane. If you are using one,
>>>>>>>> are you monitoring it to see how many connections are
>>>>>>>> actually being used? Are you sure you are using
>>>>>>>> proper resource management[1]? Even a single
>>>>>>>> code-path that leaks connections can leak them
>>>>>>>> quickly under load.
>>>>>>>>
>>>>>>>>>>> When there an issue related to broken files,
>>>>>>>>>>> this value keeps increasing, the only way to
>>>>>>>>>>> bring it down is to remove vm instance from AWS
>>>>>>>>>>> load balancer.> Which version of tomcat should
>>>>>>>>>>> I install ?
>>>>>>>> Tomcat 8.0.x hasn't been supported since its last
>>>>>>>> release on 29 June 2018. That was 8.0.53. Your
>>>>>>>> release is from 8 February 2016 and is dangerously
>>>>>>>> out of date (unless you are using the Ubuntu-packaged
>>>>>>>> version, in which case I hope they kept-up with
>>>>>>>> security patches thee past 4 years).
>>>>>>>>
>>>>>>>> -chris
>>>>>>>>
>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher
>>>>>>>>>>> Schultz < ch...@christopherschultz.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ayub,
>>>>>>>>>>>
>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases open
>>>>>>>>>>>>>> files on ubuntu AWS
>>>>>>>>>>> Which exact version of Tomcat 8.5? If you
>>>>>>>>>>> aren't running the latest version (8.5.56),
>>>>>>>>>>> please upgrade and re-test.
>>>>>>>>>>>
>>>>>>>>>>>>>> If there is slow response from db I see
>>>>>>>>>>>>>> this stack trace and the open files goes
>>>>>>>>>>>>>> high and the only way to open files go
>>>>>>>>>>>>>> down is to remove the instance from
>>>>>>>>>>>>>> Amazon load balancer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there a way to keep the open files low
>>>>>>>>>>>>>> even when Broken pipe error is thrown ?
>>>>>>>>>>> What is your evidence that file handles are
>>>>>>>>>>> being left open?
>>>>>>>>>>>
>>>>>>>>>>> Which file handles are being left open?
>>>>>>>>>>>
>>>>>>>>>>> -chris
>>>>>>>>>>>>
>>>>>>> ----------------------------------------------------------------
- ---
>>
>>
>>>>>>>
- ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For
>> additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zaP0ACgkQHPApP6U8
pFiIWw//QaDZsLlbNyYgB2XVjQzMtI0L93TnroEW+MwKnle9UNT0xYBOBk+OnFgB
nO9j1RLvbE8sROcLKv2KucZYXFxvVfx6eAp3t2OIQkaSRvjnI2Gb/ILPgI7zTWq9
/1+qjG14A1Gcxy07elRBFik2BO/XqQ6eHiRw0426Ogh7f/djFKVX6qP4TIsFWNee
shZXLt9ZA/qtmhiNyis+MdpSMenUjmZ5z9TI9iyMQirbannAxl+oVds/0rYSKZ7G
kabydTSIQpR9OBTJdTrHX79UagkulmsKSKm9f1tXaSW8WwnnEjI3tQIee5orhzHU
ZdAkMWKU9YvZC8gBLuxs6QHmHEuoxVMRGsTatnfKIdRYIPkDEqMrzpiM7AozFutS
8tFMMjQsJYBKM3CqVFr3f7WJVBYjmKMBfH6ItTwZ88RBzOMdz5Odx3JDJMQbsEH5
ylKQH9Gluqad93he0t+Uaabk6JnCdXbCpdNWyFa/fPnMp4vHFlOb7f62+UlmDT+4
DyKTl3BtDe9J6pQ0eVY4deUqFgrLCU7C4WVkOqmj9DFmFJxCMp98zXCN2h4O9lhi
OE853x4UvrPkJMPddN2pg0cJiS6Z3OQzDES1yBVbduga2LnLOJrzDpRh6NtASmNM
YC82oSScqppOgRGHKWJUrnDTyXjEeJW/eTqhnMIHHXGlwuUfVZ0=
=zQdW
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
