Chris, If some open file is owned by nginx why would it show up if I run the below command
sudo lsof -p $(cat /var/run/tomcat8.pid) On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz < [email protected]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Ayub, > > On 6/23/20 19:17, Ayub Khan wrote: > > Yes we have nginx as reverse proxy, below is the nginx config. We > > notice this issue only when there is high number of requests, > > during non peak hours we do not see this issue.> location > > /myapp/myservice{ #local machine proxy_pass > > http://localhost:8080; proxy_http_version 1.1; > > > > proxy_set_header Connection $connection_upgrade; > > proxy_set_header Upgrade $http_upgrade; > > proxy_set_header Host $host; proxy_set_header > > X-Real-IP $remote_addr; proxy_set_header > > X-Forwarded-For $proxy_add_x_forwarded_for; > > > > > > proxy_buffers 16 16k; proxy_buffer_size 32k; } > > You might want to read about tuning nginx to drop connections after a > certain period of time, number of requests, etc. Looks like either a > bug in nginx or a misconfiguration which allows connections to > stick-around like this. You may have to ask the nginx people. I have > no experience with nginx myself, while others here may have some > experience. > > > location / { # if using AWS Load balancer, this bit checks for the > > presence of the https proto flag. if regular http is found, then > > issue a redirect > to hit > > the https endpoint instead if ($http_x_forwarded_proto != 'https') > > { rewrite ^ https://$host$request_uri? permanent; } > > > > proxy_pass http://127.0.0.1:8080; proxy_http_version > > 1.1; > > > > proxy_set_header Connection $connection_upgrade; > > proxy_set_header Upgrade $http_upgrade; > > proxy_set_header Host $host; proxy_set_header > > X-Real-IP $remote_addr; proxy_set_header > > X-Forwarded-For $proxy_add_x_forwarded_for; > > > > > > proxy_buffers 16 16k; proxy_buffer_size 32k; } > > > > *below is the connector* > > > > <Connector port="8080" > > protocol="org.apache.coyote.http11.Http11NioProtocol" > > connectionTimeout="2000" maxThreads="50000" URIEncoding="UTF-8" > > redirectPort="8443" /> > > 50k threads is a LOT of threads. Do you expect to handle 50k requests > simultaneously? > > > these ports are random, I am not sure who owns the process. > > > > localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port 55866 > > is a random port. > I'm sure you'll find that 55866 is owned by nginx. netstat will tell you > . > > I think you need to look at your nginx configuration. It would also be > a great time to upgrade to a supported version of Tomcat. I would > recommend 8.5.56 or 9.0.36. > > - -chris > > > On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz < > > [email protected]> wrote: > > > > Ayub, > > > > On 6/23/20 16:23, Ayub Khan wrote: > >>>> I executed *sudo lsof -p $(cat /var/run/tomcat8.pid) *and I > >>>> saw the below output, some in CLOSE_WAIT and others in > >>>> ESTABLISHED. If there are 200 open file descriptors 160 are > >>>> in CLOSE_WAIT state. When the count for CLOSE_WAIT increases > >>>> I just have to restart tomcat. > >>>> > >>>> java 65189 tomcat8 715u IPv6 237878311 > >>>> 0t0 TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java > >>>> 65189 tomcat8 716u IPv6 237848923 0t0 > >>>> TCP localhost:http-alt->localhost:40568 (CLOSE_WAIT) > > > > These are connections from some process into Tomcat listening on > > port 8080 (that's what localhost:http-alt is). So what process owns > > the outgoing connection on port 40568 on the same host? > > > > Are you using a reverse proxy? > > > >>>> most of the open files are in CLOSE_WAIT state I do not see > >>>> anything related to database ip. > > > > Agreed. It looks like you have a reverse proxy who is losing-track > > of connections, or who is (re)opening connections when it may be > > unnecessar y. > > > > Can you share your <Connector> configuration from server.xml? > > Remember to remove any secrets. > > > > -chris > > > >>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher < > >>>> [email protected]> wrote: > >>>> > >>>>> > >>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan: > >>>>>> Felix, > >>>>>> > >>>>>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ > >>>>>> and from the > >>>>> output > >>>>>> I see majority of them are related to sockets as shown > >>>>>> below, some of > >>>>> them > >>>>>> point to the jar file of tomcat and others to the log > >>>>>> file which is > >>>>> created. > >>>>>> > >>>>>> socket:[2084570754] socket:[2084579487] > >>>>>> socket:[2084578478] socket:[2084570167] > >>>>> > >>>>> Can you try the other command (lsof -p $(cat > >>>>> ...tomcat.pid))? It should give a bit more details on the > >>>>> used sockets that the proc directory. > >>>>> > >>>>> Felix > >>>>> > >>>>>> > >>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan: > >>>>>>>> Chris, > >>>>>>>> > >>>>>>>> I am using HikariCP for connection pooling. If the > >>>>>>>> database is leaking connections then I should see > >>>>>>>> connection not available exception. > >>>>>>>> > >>>>>>>> How do I find out which file descriptors are leaking > >>>>>>>> ? these are not > >>>>>>> files > >>>>>>>> open on disk as there is no explicit disk file I/O in > >>>>>>>> this application. > >>>>>>>> > >>>>>>>> I just use the below command to check for open file > >>>>>>>> descriptors: > >>>>>>>> > >>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | > >>>>>>>> wc -l" > >>>>>>> You could have a look at the name of the files in the > >>>>>>> pids proc > >>>>> directory. > >>>>>>> > >>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ > >>>>>>> > >>>>>>> Or you could use the tool lsof to find the open file > >>>>>>> descriptors. > >>>>>>> > >>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid) > >>>>>>> > >>>>>>> For both calls you should first change to the uid of > >>>>>>> the tomcat user or use sudo as in your example. > >>>>>>> > >>>>>>> Felix > >>>>>>> > >>>>>>>> Thanks and Regards Ayub > >>>>>>>> > >>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz > >>>>>>>> < [email protected]> wrote: > >>>>>>>> > >>>>>>>> Ayub, > >>>>>>>> > >>>>>>>> On 6/20/20 11:51, Ayub Khan wrote: > >>>>>>>>>>> Sorry we are using 8.0.32 version of tomcat. > >>>>>>>>>>> > >>>>>>>>>>> below is the configuration: > >>>>>>>>>>> > >>>>>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) > >>>>>>>>>>> Server built: Jan 24 2020 16:24:30 UTC Server > >>>>>>>>>>> number: 8.0.32.0 OS Name: Linux OS Version: > >>>>>>>>>>> 4.4.0-1087-aws Architecture: amd64 JVM > >>>>>>>>>>> Version: 1.8.0_181-b13 JVM Vendor: Oracle > >>>>>>>>>>> Corporation > >>>>>>>>>>> > >>>>>>>>>>> I use the below command to check the file > >>>>>>>>>>> descriptors: > >>>>>>>>>>> > >>>>>>>>>>> watch "sudo ls /proc/`cat > >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l" > >>>>>>>> So you know there is some kind of increase in > >>>>>>>> file-handle use, but you don't know what types of > >>>>>>>> file handles are increasing, right? > >>>>>>>> > >>>>>>>> Can you try to find out which kinds of file handles > >>>>>>>> are increasing? > >>>>>>>> > >>>>>>>> I have a sneaking suspicion that it's your database > >>>>>>>> connections and not actually files open on the disk. > >>>>>>>> > >>>>>>>> Are you using a database connection pool? If not, > >>>>>>>> you should really use one and limit the number of > >>>>>>>> connections to something sane. If you are using one, > >>>>>>>> are you monitoring it to see how many connections are > >>>>>>>> actually being used? Are you sure you are using > >>>>>>>> proper resource management[1]? Even a single > >>>>>>>> code-path that leaks connections can leak them > >>>>>>>> quickly under load. > >>>>>>>> > >>>>>>>>>>> When there an issue related to broken files, > >>>>>>>>>>> this value keeps increasing, the only way to > >>>>>>>>>>> bring it down is to remove vm instance from AWS > >>>>>>>>>>> load balancer.> Which version of tomcat should > >>>>>>>>>>> I install ? > >>>>>>>> Tomcat 8.0.x hasn't been supported since its last > >>>>>>>> release on 29 June 2018. That was 8.0.53. Your > >>>>>>>> release is from 8 February 2016 and is dangerously > >>>>>>>> out of date (unless you are using the Ubuntu-packaged > >>>>>>>> version, in which case I hope they kept-up with > >>>>>>>> security patches thee past 4 years). > >>>>>>>> > >>>>>>>> -chris > >>>>>>>> > >>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher > >>>>>>>>>>> Schultz < [email protected]> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Ayub, > >>>>>>>>>>> > >>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote: > >>>>>>>>>>>>>> tomcat 8.5 broken pipe increases open > >>>>>>>>>>>>>> files on ubuntu AWS > >>>>>>>>>>> Which exact version of Tomcat 8.5? If you > >>>>>>>>>>> aren't running the latest version (8.5.56), > >>>>>>>>>>> please upgrade and re-test. > >>>>>>>>>>> > >>>>>>>>>>>>>> If there is slow response from db I see > >>>>>>>>>>>>>> this stack trace and the open files goes > >>>>>>>>>>>>>> high and the only way to open files go > >>>>>>>>>>>>>> down is to remove the instance from > >>>>>>>>>>>>>> Amazon load balancer. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Is there a way to keep the open files low > >>>>>>>>>>>>>> even when Broken pipe error is thrown ? > >>>>>>>>>>> What is your evidence that file handles are > >>>>>>>>>>> being left open? > >>>>>>>>>>> > >>>>>>>>>>> Which file handles are being left open? > >>>>>>>>>>> > >>>>>>>>>>> -chris > >>>>>>>>>>>> > >>>>>>> ---------------------------------------------------------------- > - --- > >> > >> > >>>>>>> > - --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] For > >> additional commands, e-mail: [email protected] > >> > >> > > > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zaP0ACgkQHPApP6U8 > pFiIWw//QaDZsLlbNyYgB2XVjQzMtI0L93TnroEW+MwKnle9UNT0xYBOBk+OnFgB > nO9j1RLvbE8sROcLKv2KucZYXFxvVfx6eAp3t2OIQkaSRvjnI2Gb/ILPgI7zTWq9 > /1+qjG14A1Gcxy07elRBFik2BO/XqQ6eHiRw0426Ogh7f/djFKVX6qP4TIsFWNee > shZXLt9ZA/qtmhiNyis+MdpSMenUjmZ5z9TI9iyMQirbannAxl+oVds/0rYSKZ7G > kabydTSIQpR9OBTJdTrHX79UagkulmsKSKm9f1tXaSW8WwnnEjI3tQIee5orhzHU > ZdAkMWKU9YvZC8gBLuxs6QHmHEuoxVMRGsTatnfKIdRYIPkDEqMrzpiM7AozFutS > 8tFMMjQsJYBKM3CqVFr3f7WJVBYjmKMBfH6ItTwZ88RBzOMdz5Odx3JDJMQbsEH5 > ylKQH9Gluqad93he0t+Uaabk6JnCdXbCpdNWyFa/fPnMp4vHFlOb7f62+UlmDT+4 > DyKTl3BtDe9J6pQ0eVY4deUqFgrLCU7C4WVkOqmj9DFmFJxCMp98zXCN2h4O9lhi > OE853x4UvrPkJMPddN2pg0cJiS6Z3OQzDES1yBVbduga2LnLOJrzDpRh6NtASmNM > YC82oSScqppOgRGHKWJUrnDTyXjEeJW/eTqhnMIHHXGlwuUfVZ0= > =zQdW > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- -------------------------------------------------------------------- Sun Certified Enterprise Architect 1.5 Sun Certified Java Programmer 1.4 Microsoft Certified Systems Engineer 2000 http://in.linkedin.com/pub/ayub-khan/a/811/b81 mobile:+966-502674604 ---------------------------------------------------------------------- It is proved that Hard Work and kowledge will get you close but attitude will get you there. However, it's the Love of God that will put you over the top!!
