Chris,
Was just thinking if the file descriptors belonged to nginx why do they
disappear as soon as I restart tomcat ? I tried restarting nginx and the
open file descriptors don't disappear.
When I execute lsof -p <nginx pid> I do not see file descriptors in close
wait state
On Wed, 24 Jun 2020, 20:32 Ayub Khan, <ayub...@gmail.com> wrote:
> Chris,
>
> Ok, I will investigate nginx side as well. Thank you for the pointers
>
> On Wed, 24 Jun 2020, 19:45 Christopher Schultz, <
> ch...@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>>
>> Ayub,
>>
>> On 6/24/20 11:05, Ayub Khan wrote:
>> > If some open file is owned by nginx why would it show up if I run
>> > the below command> sudo lsof -p $(cat /var/run/tomcat8.pid)
>>
>> Because network connections have two ends.
>>
>> - -chris
>>
>> > On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
>> > ch...@christopherschultz.net> wrote:
>> >
>> > Ayub,
>> >
>> > On 6/23/20 19:17, Ayub Khan wrote:
>> >>>> Yes we have nginx as reverse proxy, below is the nginx
>> >>>> config. We notice this issue only when there is high number
>> >>>> of requests, during non peak hours we do not see this issue.>
>> >>>> location /myapp/myservice{ #local machine proxy_pass
>> >>>> http://localhost:8080; proxy_http_version 1.1;
>> >>>>
>> >>>> proxy_set_header Connection $connection_upgrade;
>> >>>> proxy_set_header Upgrade $http_upgrade;
>> >>>> proxy_set_header Host $host;
>> >>>> proxy_set_header X-Real-IP $remote_addr;
>> >>>> proxy_set_header X-Forwarded-For
>> >>>> $proxy_add_x_forwarded_for;
>> >>>>
>> >>>>
>> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>> >
>> > You might want to read about tuning nginx to drop connections after
>> > a certain period of time, number of requests, etc. Looks like
>> > either a bug in nginx or a misconfiguration which allows
>> > connections to stick-around like this. You may have to ask the
>> > nginx people. I have no experience with nginx myself, while others
>> > here may have some experience.
>> >
>> >>>> location / { # if using AWS Load balancer, this bit checks
>> >>>> for the presence of the https proto flag. if regular http is
>> >>>> found, then issue a redirect
>> > to hit
>> >>>> the https endpoint instead if ($http_x_forwarded_proto !=
>> >>>> 'https') { rewrite ^ https://$host$request_uri? permanent; }
>> >>>>
>> >>>> proxy_pass http://127.0.0.1:8080;
>> >>>> proxy_http_version 1.1;
>> >>>>
>> >>>> proxy_set_header Connection $connection_upgrade;
>> >>>> proxy_set_header Upgrade $http_upgrade;
>> >>>> proxy_set_header Host $host;
>> >>>> proxy_set_header X-Real-IP $remote_addr;
>> >>>> proxy_set_header X-Forwarded-For
>> >>>> $proxy_add_x_forwarded_for;
>> >>>>
>> >>>>
>> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>> >>>>
>> >>>> *below is the connector*
>> >>>>
>> >>>> <Connector port="8080"
>> >>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> >>>> connectionTimeout="2000" maxThreads="50000"
>> >>>> URIEncoding="UTF-8" redirectPort="8443" />
>> >
>> > 50k threads is a LOT of threads. Do you expect to handle 50k
>> > requests simultaneously?
>> >
>> >>>> these ports are random, I am not sure who owns the process.
>> >>>>
>> >>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port
>> >>>> 55866 is a random port.
>> > I'm sure you'll find that 55866 is owned by nginx. netstat will
>> > tell you .
>> >
>> > I think you need to look at your nginx configuration. It would also
>> > be a great time to upgrade to a supported version of Tomcat. I
>> > would recommend 8.5.56 or 9.0.36.
>> >
>> > -chris
>> >
>> >>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
>> >>>> ch...@christopherschultz.net> wrote:
>> >>>>
>> >>>> Ayub,
>> >>>>
>> >>>> On 6/23/20 16:23, Ayub Khan wrote:
>> >>>>>>> I executed *sudo lsof -p $(cat /var/run/tomcat8.pid)
>> >>>>>>> *and I saw the below output, some in CLOSE_WAIT and
>> >>>>>>> others in ESTABLISHED. If there are 200 open file
>> >>>>>>> descriptors 160 are in CLOSE_WAIT state. When the count
>> >>>>>>> for CLOSE_WAIT increases I just have to restart
>> >>>>>>> tomcat.
>> >>>>>>>
>> >>>>>>> java 65189 tomcat8 715u IPv6
>> >>>>>>> 237878311 0t0 TCP localhost:http-alt->localhost:43760
>> >>>>>>> (CLOSE_WAIT) java 65189 tomcat8 716u IPv6
>> >>>>>>> 237848923 0t0 TCP
>> >>>>>>> localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>> >>>>
>> >>>> These are connections from some process into Tomcat listening
>> >>>> on port 8080 (that's what localhost:http-alt is). So what
>> >>>> process owns the outgoing connection on port 40568 on the
>> >>>> same host?
>> >>>>
>> >>>> Are you using a reverse proxy?
>> >>>>
>> >>>>>>> most of the open files are in CLOSE_WAIT state I do not
>> >>>>>>> see anything related to database ip.
>> >>>>
>> >>>> Agreed. It looks like you have a reverse proxy who is
>> >>>> losing-track of connections, or who is (re)opening
>> >>>> connections when it may be unnecessar y.
>> >>>>
>> >>>> Can you share your <Connector> configuration from
>> >>>> server.xml? Remember to remove any secrets.
>> >>>>
>> >>>> -chris
>> >>>>
>> >>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
>> >>>>>>> felix.schumac...@internetallee.de> wrote:
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
>> >>>>>>>>> Felix,
>> >>>>>>>>>
>> >>>>>>>>> I executed ls -l /proc/$(cat
>> >>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the
>> >>>>>>>> output
>> >>>>>>>>> I see majority of them are related to sockets as
>> >>>>>>>>> shown below, some of
>> >>>>>>>> them
>> >>>>>>>>> point to the jar file of tomcat and others to the
>> >>>>>>>>> log file which is
>> >>>>>>>> created.
>> >>>>>>>>>
>> >>>>>>>>> socket:[2084570754] socket:[2084579487]
>> >>>>>>>>> socket:[2084578478] socket:[2084570167]
>> >>>>>>>>
>> >>>>>>>> Can you try the other command (lsof -p $(cat
>> >>>>>>>> ...tomcat.pid))? It should give a bit more details on
>> >>>>>>>> the used sockets that the proc directory.
>> >>>>>>>>
>> >>>>>>>> Felix
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
>> >>>>>>>>> felix.schumac...@internetallee.de> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>> >>>>>>>>>>> Chris,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I am using HikariCP for connection pooling. If
>> >>>>>>>>>>> the database is leaking connections then I
>> >>>>>>>>>>> should see connection not available exception.
>> >>>>>>>>>>>
>> >>>>>>>>>>> How do I find out which file descriptors are
>> >>>>>>>>>>> leaking ? these are not
>> >>>>>>>>>> files
>> >>>>>>>>>>> open on disk as there is no explicit disk file
>> >>>>>>>>>>> I/O in this application.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I just use the below command to check for open
>> >>>>>>>>>>> file descriptors:
>> >>>>>>>>>>>
>> >>>>>>>>>>> watch "sudo ls /proc/`cat
>> >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>> >>>>>>>>>> You could have a look at the name of the files in
>> >>>>>>>>>> the pids proc
>> >>>>>>>> directory.
>> >>>>>>>>>>
>> >>>>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>> >>>>>>>>>>
>> >>>>>>>>>> Or you could use the tool lsof to find the open
>> >>>>>>>>>> file descriptors.
>> >>>>>>>>>>
>> >>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
>> >>>>>>>>>>
>> >>>>>>>>>> For both calls you should first change to the uid
>> >>>>>>>>>> of the tomcat user or use sudo as in your
>> >>>>>>>>>> example.
>> >>>>>>>>>>
>> >>>>>>>>>> Felix
>> >>>>>>>>>>
>> >>>>>>>>>>> Thanks and Regards Ayub
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher
>> >>>>>>>>>>> Schultz < ch...@christopherschultz.net> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Ayub,
>> >>>>>>>>>>>
>> >>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
>> >>>>>>>>>>>>>> Sorry we are using 8.0.32 version of
>> >>>>>>>>>>>>>> tomcat.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> below is the configuration:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Server version: Apache Tomcat/8.0.32
>> >>>>>>>>>>>>>> (Ubuntu) Server built: Jan 24 2020
>> >>>>>>>>>>>>>> 16:24:30 UTC Server number: 8.0.32.0 OS
>> >>>>>>>>>>>>>> Name: Linux OS Version: 4.4.0-1087-aws
>> >>>>>>>>>>>>>> Architecture: amd64 JVM Version:
>> >>>>>>>>>>>>>> 1.8.0_181-b13 JVM Vendor: Oracle
>> >>>>>>>>>>>>>> Corporation
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I use the below command to check the
>> >>>>>>>>>>>>>> file descriptors:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> watch "sudo ls /proc/`cat
>> >>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>> >>>>>>>>>>> So you know there is some kind of increase in
>> >>>>>>>>>>> file-handle use, but you don't know what types
>> >>>>>>>>>>> of file handles are increasing, right?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Can you try to find out which kinds of file
>> >>>>>>>>>>> handles are increasing?
>> >>>>>>>>>>>
>> >>>>>>>>>>> I have a sneaking suspicion that it's your
>> >>>>>>>>>>> database connections and not actually files
>> >>>>>>>>>>> open on the disk.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Are you using a database connection pool? If
>> >>>>>>>>>>> not, you should really use one and limit the
>> >>>>>>>>>>> number of connections to something sane. If you
>> >>>>>>>>>>> are using one, are you monitoring it to see how
>> >>>>>>>>>>> many connections are actually being used? Are
>> >>>>>>>>>>> you sure you are using proper resource
>> >>>>>>>>>>> management[1]? Even a single code-path that
>> >>>>>>>>>>> leaks connections can leak them quickly under
>> >>>>>>>>>>> load.
>> >>>>>>>>>>>
>> >>>>>>>>>>>>>> When there an issue related to broken
>> >>>>>>>>>>>>>> files, this value keeps increasing, the
>> >>>>>>>>>>>>>> only way to bring it down is to remove vm
>> >>>>>>>>>>>>>> instance from AWS load balancer.> Which
>> >>>>>>>>>>>>>> version of tomcat should I install ?
>> >>>>>>>>>>> Tomcat 8.0.x hasn't been supported since its
>> >>>>>>>>>>> last release on 29 June 2018. That was 8.0.53.
>> >>>>>>>>>>> Your release is from 8 February 2016 and is
>> >>>>>>>>>>> dangerously out of date (unless you are using
>> >>>>>>>>>>> the Ubuntu-packaged version, in which case I
>> >>>>>>>>>>> hope they kept-up with security patches thee
>> >>>>>>>>>>> past 4 years).
>> >>>>>>>>>>>
>> >>>>>>>>>>> -chris
>> >>>>>>>>>>>
>> >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM
>> >>>>>>>>>>>>>> Christopher Schultz <
>> >>>>>>>>>>>>>> ch...@christopherschultz.net> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Ayub,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>> >>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases
>> >>>>>>>>>>>>>>>>> open files on ubuntu AWS
>> >>>>>>>>>>>>>> Which exact version of Tomcat 8.5? If
>> >>>>>>>>>>>>>> you aren't running the latest version
>> >>>>>>>>>>>>>> (8.5.56), please upgrade and re-test.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> If there is slow response from db I
>> >>>>>>>>>>>>>>>>> see this stack trace and the open
>> >>>>>>>>>>>>>>>>> files goes high and the only way to
>> >>>>>>>>>>>>>>>>> open files go down is to remove the
>> >>>>>>>>>>>>>>>>> instance from Amazon load
>> >>>>>>>>>>>>>>>>> balancer.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Is there a way to keep the open
>> >>>>>>>>>>>>>>>>> files low even when Broken pipe
>> >>>>>>>>>>>>>>>>> error is thrown ?
>> >>>>>>>>>>>>>> What is your evidence that file handles
>> >>>>>>>>>>>>>> are being left open?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Which file handles are being left open?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> -chris
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>> -------------------------------------------------------------
>> - ---
>> >
>> >>>>>>>>>>
>> - ---
>> >>>>>
>> >>>>>
>> >>>>>>>>>>
>> > ---------------------------------------------------------------------
>> >>>>>
>> >
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For
>> >>>>> additional commands, e-mail: users-h...@tomcat.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>
>> >> ---------------------------------------------------------------------
>> >>
>> >>
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >>
>> >>
>> >
>> -----BEGIN PGP SIGNATURE-----
>> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>>
>> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zgOIACgkQHPApP6U8
>> pFhnWQ//SBTsCR/g8cDgrC+ju6x3BA8kpEmNbA4BP5fY8I7nrDoiegsrZDbRsoWI
>> rNlH3i3pLlonTbbLQN/6AeWItmgM88bupy007cEcNrNivztkOPscmFOmPZP7lNaD
>> JfgXQvTF1q8LFL+R120GGVR+OKM06dZ4rYbWbu+Ju46lYX6q2iVlb+n8XmmPBIPg
>> 9dEE4kXNc+A3ZI+3JxLoWnxnDSxpG2Od2WDwt+dLV0E7XkHJCMKGqtzbPWcUtaUR
>> nh9rywiO4ODk57VbtU426yB6OrNxbwj2OOKpETKVfLpUsdwzvgEZgzbfeUqPpR4p
>> OpIXxjgBeGFhk+ssafvqOMIMI4EO/m/+uttyfQhAh05ozP99NLTZXbDcnY9gFEeY
>> Rk7Z7qnGey3erk83dfR3X56zovuiS6k6vHKLZcp6Js5ta1cQa9D8hOjxcVRaDWlz
>> AeKNEiwuuIo2tfsl2OCAxkoNpl+dooE2aXcvAWbrI7pGXMWwANzEGw4NmIUTOvpS
>> tjR4OqaOKiRo1TN66JZVeol0ZFlaz9InxHTEasHrdn80YJGMIFL7UmR6koZGUE2G
>> 7k0yukOUQCDLCPc3HwCCfOw/rkygLtxKShMEcZDp2gH3AZo0z1AKBDGWyi7XyMKa
>> MO4DkMHhp28aRZ84Dtsyvxh/HuxBt9Vl2hj/oOkBMVbfrIT+3bg=
>> =q6xp
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
