Hi, Yes; that is what I get and as you can see among 8028 open files, 7474 are for TCPv6 sockets: java 130244 root 7805u sock 0,9 0t0 12294251 protocol: TCPv6
# ps -ef | grep tomcat root 130244 1 1 11:01 ? 00:06:20 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto tomcat #lsof -p 130244 | wc -l 8028 #lsof -p 130244 | grep "protocol: TCPv6"| wc -l 7474 Thanks -----Original Message----- From: André Warnier (tomcat/perl) <a...@ice-sa.com> Sent: Thursday, May 20, 2021 4:19 PM To: users@tomcat.apache.org Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 Hi. According to the list below, you have 2 java (JVM) processes running on your system. One (PID = 130244) is the JVM which runs tomcat. This is visible when you look at the whole command-line. The other (PID = 130516) runs ElasticSearch, which I believe is not relevant here. So you should run lsof with the "-p 130244" option, to show only the files opened by the tomcat JVM. To show the current Tomcat JVM PID, do e.g. : ps -ef | grep "apache-tomcat" (or anything else unique in that line) On 20.05.2021 21:00, Yeggy Javadi wrote: > Hi Chris, > Please indicate how to show lsof or netstat to *just the JVM* process. > > Below is the list of running processes on my server: > UID PID PPID C STIME TTY TIME CMD > root 1 0 0 May07 ? 00:00:14 /usr/lib/systemd/systemd > --switched-root --system --deserialize 17 > root 2 0 0 May07 ? 00:00:00 [kthreadd] > root 3 2 0 May07 ? 00:00:00 [rcu_gp] > root 4 2 0 May07 ? 00:00:00 [rcu_par_gp] > root 6 2 0 May07 ? 00:00:00 [kworker/0:0H-kblockd] > root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq] > root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0] > root 10 2 0 May07 ? 00:02:18 [rcu_sched] > root 11 2 0 May07 ? 00:00:00 [migration/0] > root 12 2 0 May07 ? 00:00:00 [watchdog/0] > root 13 2 0 May07 ? 00:00:00 [cpuhp/0] > root 14 2 0 May07 ? 00:00:00 [cpuhp/1] > root 15 2 0 May07 ? 00:00:00 [watchdog/1] > root 16 2 0 May07 ? 00:00:00 [migration/1] > root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1] > root 19 2 0 May07 ? 00:00:00 [kworker/1:0H-kblockd] > root 20 2 0 May07 ? 00:00:00 [cpuhp/2] > root 21 2 0 May07 ? 00:00:00 [watchdog/2] > root 22 2 0 May07 ? 00:00:00 [migration/2] > root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2] > root 25 2 0 May07 ? 00:00:00 [kworker/2:0H-kblockd] > root 26 2 0 May07 ? 00:00:00 [cpuhp/3] > root 27 2 0 May07 ? 00:00:00 [watchdog/3] > root 28 2 0 May07 ? 00:00:00 [migration/3] > root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3] > root 31 2 0 May07 ? 00:00:00 [kworker/3:0H-kblockd] > root 32 2 0 May07 ? 00:00:00 [cpuhp/4] > root 33 2 0 May07 ? 00:00:00 [watchdog/4] > root 34 2 0 May07 ? 00:00:00 [migration/4] > root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4] > root 37 2 0 May07 ? 00:00:00 [kworker/4:0H-kblockd] > root 38 2 0 May07 ? 00:00:00 [cpuhp/5] > root 39 2 0 May07 ? 00:00:00 [watchdog/5] > root 40 2 0 May07 ? 00:00:00 [migration/5] > root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5] > root 43 2 0 May07 ? 00:00:00 [kworker/5:0H-kblockd] > root 44 2 0 May07 ? 00:00:00 [cpuhp/6] > root 45 2 0 May07 ? 00:00:00 [watchdog/6] > root 46 2 0 May07 ? 00:00:00 [migration/6] > root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6] > root 49 2 0 May07 ? 00:00:00 [kworker/6:0H-kblockd] > root 50 2 0 May07 ? 00:00:00 [cpuhp/7] > root 51 2 0 May07 ? 00:00:00 [watchdog/7] > root 52 2 0 May07 ? 00:00:00 [migration/7] > root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7] > root 55 2 0 May07 ? 00:00:00 [kworker/7:0H-kblockd] > root 57 2 0 May07 ? 00:00:00 [kdevtmpfs] > root 58 2 0 May07 ? 00:00:00 [netns] > root 59 2 0 May07 ? 00:00:00 [kauditd] > root 62 2 0 May07 ? 00:00:00 [khungtaskd] > root 63 2 0 May07 ? 00:00:00 [oom_reaper] > root 64 2 0 May07 ? 00:00:00 [writeback] > root 65 2 0 May07 ? 00:00:00 [kcompactd0] > root 66 2 0 May07 ? 00:00:00 [ksmd] > root 67 2 0 May07 ? 00:00:02 [khugepaged] > root 68 2 0 May07 ? 00:00:00 [crypto] > root 69 2 0 May07 ? 00:00:00 [kintegrityd] > root 70 2 0 May07 ? 00:00:00 [kblockd] > root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq] > root 72 2 0 May07 ? 00:00:00 [md] > root 73 2 0 May07 ? 00:00:00 [edac-poller] > root 74 2 0 May07 ? 00:00:00 [watchdogd] > root 92 2 0 May07 ? 00:00:00 [kswapd0] > root 188 2 0 May07 ? 00:00:00 [kthrotld] > root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp] > root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp] > root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp] > root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp] > root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp] > root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp] > root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp] > root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp] > root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp] > root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp] > root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp] > root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp] > root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp] > root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp] > root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp] > root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp] > root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp] > root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp] > root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp] > root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp] > root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp] > root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp] > root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp] > root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp] > root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp] > root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp] > root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp] > root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp] > root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp] > root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp] > root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp] > root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp] > root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm] > root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd] > root 224 2 0 May07 ? 00:00:00 [kaluad] > root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf] > root 227 2 0 May07 ? 00:00:00 [kstrp] > root 508 2 0 May07 ? 00:00:00 [mpt_poll_0] > root 509 2 0 May07 ? 00:00:00 [ata_sff] > root 510 2 0 May07 ? 00:00:00 [mpt/0] > root 512 2 0 May07 ? 00:00:00 [scsi_eh_0] > root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0] > root 517 2 0 May07 ? 00:00:00 [scsi_eh_1] > root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1] > root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx] > root 525 2 0 May07 ? 00:00:00 [ttm_swap] > root 531 2 0 May07 ? 00:00:00 [scsi_eh_2] > root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2] > root 540 2 0 May07 ? 00:00:01 [kworker/7:1H-kblockd] > root 541 2 0 May07 ? 00:00:01 [kworker/5:1H-kblockd] > root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8] > root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8] > root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > root 613 2 0 May07 ? 00:00:01 [kworker/6:1H-kblockd] > root 650 2 0 May07 ? 00:00:01 [kworker/4:1H-kblockd] > root 659 2 0 May07 ? 00:00:02 [kworker/3:1H-kblockd] > root 660 2 0 May07 ? 00:00:02 [kworker/2:1H-kblockd] > root 662 2 0 May07 ? 00:00:01 [kworker/0:1H-kblockd] > root 679 1 0 May07 ? 00:00:03 > /usr/lib/systemd/systemd-journald > root 682 2 0 May07 ? 00:00:02 [kworker/1:1H-kblockd] > root 716 1 0 May07 ? 00:00:01 > /usr/lib/systemd/systemd-udevd > root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8] > root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8] > root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8] > root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8] > root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > root 776 1 0 May07 ? 00:00:01 /sbin/auditd > root 790 2 0 May07 ? 00:00:00 [rpciod] > root 791 2 0 May07 ? 00:00:00 [kworker/u17:0] > root 792 2 0 May07 ? 00:00:00 [xprtiod] > dbus 883 1 0 May07 ? 00:00:03 /usr/bin/dbus-daemon > --system --address=systemd: --nofork --nopidfile --systemd-activation > --syslog-only > root 885 1 0 May07 ? 00:02:42 /usr/sbin/NetworkManager > --no-daemon > root 887 1 0 May07 ? 00:00:44 /usr/sbin/irqbalance > --foreground > root 888 1 0 May07 ? 00:00:00 /usr/bin/VGAuthService -s > root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd > root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i > --logger=files > chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd > rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f > --fill-watermark=0 > root 946 890 0 May07 ? 00:00:06 /usr/libexec/sssd/sssd_be > --domain implicit_files --uid 0 --gid 0 --logger=files > root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f -d2 > root 962 890 0 May07 ? 00:00:12 > /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files > root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy -D > root 970 1 0 May07 ? 00:00:02 > /usr/lib/systemd/systemd-logind > root 971 2 0 May07 ? 00:00:00 [iscsi_eh] > root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd -n > root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n > root 6223 1 0 May07 ? 00:07:27 > /usr/local/vtl/bin/sanidmgr > root 6271 2 0 May07 ? 00:00:03 [bksc_exec] > root 6272 2 0 May07 ? 00:00:03 [bksc_done] > root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3] > root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3] > root 6460 2 0 May07 ? 00:00:00 [kdownstream] > root 6461 2 0 May07 ? 00:00:07 [msgfilter] > root 6463 2 0 May07 ? 00:00:12 [iocore] > root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr] > root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr] > root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0 > root 6523 2 0 May07 ? 00:01:50 [trudpd] > root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker] > root 6541 2 0 May07 ? 00:00:00 [fsnalias_check] > root 6551 2 0 May07 ? 00:00:00 [misc_thread] > root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen] > root 6554 2 0 May07 ? 00:00:00 [listen_handle] > root 6557 2 0 May07 ? 00:00:00 [ipstord] > root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq] > root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4] > root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4] > root 6654 1 0 May07 ? 00:00:00 > /usr/local/vtl/bin/ipstorcomm -d 0 > root 6655 6654 0 May07 ? 00:05:18 > /usr/local/vtl/bin/ipstorcomm -d 0 > root 6656 6655 0 May07 ? 00:00:36 > /usr/local/vtl/bin/ipstorcomm -d 0 > root 6693 1 0 May07 ? 00:00:00 rdesirkey > root 6699 1 0 May07 ? 00:00:12 ikmd > root 6721 1 0 May07 ? 00:00:35 > /usr/local/vtl/bin/iscliproxy > root 6747 1 0 May07 ? 00:00:40 > /usr/local/vtl/bin/ipstorlog > root 6780 1 0 May07 ? 00:04:50 /usr/local/vtl/bin/snmpd > root 7722 1 0 May07 ? 00:00:00 > /usr/local/vtl/bin/ipstorsm > root 7723 7722 0 May07 ? 01:08:12 > /usr/local/vtl/bin/ipstorsm > root 7901 1 0 May07 ? 00:00:00 /usr/lib/systemd/systemd > --user > root 7904 7901 0 May07 ? 00:00:00 (sd-pam) > root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p -- \u > --noclear tty1 linux > root 14199 1 0 May08 ? 00:00:01 sshd: root [priv] > root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0 > root 14202 14201 0 May08 pts/0 00:00:00 -bash > root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D > -oCiphers=aes256-...@openssh.com,chacha20-poly1...@openssh.com,aes256-ctr,aes256-cbc,aes128-...@openssh.com,aes128-ctr,aes128-cbc > > -oMACs=hmac-sha2-256-...@openssh.com,hmac-sha1-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha2-256,hmac-sha1,umac-...@openssh.com,hmac-sha2-512 > > -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1- > > -oKexAlgorithms=curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 > > -oHostKeyAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-...@openssh.com,ssh-ed25519,ssh-ed25519-cert-...@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-...@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-...@openssh.com,ssh-rsa,ssh-rsa-cert-...@openssh.com > > -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-...@openssh.com,ssh-ed25519,ssh-ed25519-cert-...@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-...@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-...@openssh.com,ssh-rsa,ssh-rsa-cert-...@openssh.com > > -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa > root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events] > root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events] > root 128108 2 0 08:40 ? 00:00:00 > [kworker/4:1-events_power_efficient] > root 128360 2 0 09:10 ? 00:00:00 > [kworker/4:2-cgroup_destroy] > root 128430 2 0 09:30 ? 00:00:00 [kworker/6:0-mm_percpu_wq] > root 128738 2 0 11:00 ? 00:00:00 [kworker/3:3-mm_percpu_wq] > root 128761 2 0 11:00 ? 00:00:00 [kworker/2:1-mm_percpu_wq] > root 130244 1 1 11:01 ? 00:03:50 /usr/local/jre/bin/java > -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 > -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dignore.endorsed.dirs= -classpath > /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar > -Dcatalina.base=/usr/local/apache-tomcat > -Dcatalina.home=/usr/local/apache-tomcat > -Djava.io.tmpdir=/usr/local/apache-tomcat/temp > org.apache.catalina.startup.Bootstrap start > elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g -Xmx8g > -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 > -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch > -cp > :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* > -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid > -Des.default.path.home=/usr/share/elasticsearch > -Des.default.path.logs=/var/log/elasticsearch > -Des.default.path.data=/var/lib/elasticsearch > -Des.default.config=/etc/elasticsearch/elasticsearch.yml > -Des.default.path.conf=/etc/elasticsearch > org.elasticsearch.bootstrap.Elasticsearch > postgres 130548 1 0 11:01 ? 00:00:00 > /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data > postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger process > postgres 130563 130548 0 11:01 ? 00:00:00 postgres: checkpointer > process > postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer process > postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal writer > process > postgres 130566 130548 0 11:01 ? 00:00:00 postgres: autovacuum > launcher process > postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats collector > process > root 130570 1 0 11:01 ? 00:00:01 > /usr/local/freestor/bin/fmsd > postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres > freestor 127.0.0.1(42660) idle > postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres > freestor 127.0.0.1(42662) idle > postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres > freestor 127.0.0.1(42664) idle > root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2] > root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2] > root 131347 2 0 13:49 ? 00:00:00 [kworker/5:1-mm_percpu_wq] > root 131364 2 0 13:56 ? 00:00:00 > [kworker/u16:2-events_unbound] > root 131401 2 0 14:05 ? 00:00:00 > [kworker/u16:1-events_unbound] > root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2] > root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events] > root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0] > root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events] > root 131515 2 0 14:49 ? 00:00:00 > [kworker/1:0-events_freezable] > root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0] > root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv] > root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1 > root 131530 131529 0 14:52 pts/1 00:00:00 -bash > root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef > > Thanks > -----Original Message----- > From: Christopher Schultz <ch...@christopherschultz.net> > Sent: Thursday, May 20, 2021 12:49 PM > To: users@tomcat.apache.org > Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 > > Yeggy, > > On 5/19/21 19:37, Yeggy Javadi wrote: >> Hi Chris, >> Thanks for your email. Below are replies to your questions: >> >> 1. Did you upgrade anything recently (like Java VM)? >> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6. >> >> 2. What is error you are seeing? A full stack trace would be helpful. >> [YJ] The application error can occur in any place with the (Too many open >> files) error when the limit of open files for tomcat has reached (262144 in >> my environment), for example, I can get a connectivity error when pulling >> info from a server as below : >> >> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - >> Error : The XML configuration file failed to be retrieved for server >> 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for >> server=10.1.6.25 error=java.io.FileNotFoundException: >> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3. >> 2.10.RELEASE.jar (Too many open files) restatus=-1 output= >> >> 3. What is your <Connector> configuration? >> [YJ] It is as below in server.xml: >> >> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1" >> maxThreads="150" >> minSpareThreads="25" maxSpareThreads="75" >> enableLookups="false" acceptCount="100" >> connectionTimeout="20000" disableUploadTimeout="true" >> compression="on" >> compressionMinSize="2048" >> noCompressionUserAgents="gozilla, traviata" >> compressableMimeType="text/html,text/xml,application/json" >> redirectPort="443" /> >> ........... >> <Connector address="0.0.0.0" port="443" >> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true" >> maxThreads="150" scheme="https" secure="true" >> clientAuth="false" sslEnabledProtocols="TLSv1.2" >> >> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA" >> keystoreFile="conf/freestor" keystorePass="freestor"/> >> >> <!-- Define an AJP 1.3 Connector on port 8009 --> >> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3" >> redirectPort="443" secretRequired="false" /> > > You have lots of settings on the HTTP connector but not the HTTPS one. > Is that intentional? Which one are you using? Or both? > > Are you using AJP? If not, disable the connetor entirely. Definitely never > enable it on 0.0.0.0 unless you really know what you are doing. > >> 4. Are you counting all the open files for a single process or all of the >> sub-processes which represent the threads of the main process? >> Different kernel versions count things differently. >> [YJ] I am just getting the process ID of tomcat and count open files >> for that process > > Ok. > >> 5. Running lsof, netstat, etc. can you see if some large number of those >> sockets are bound to any specific port (listen or connect)? >> [YJ] Here is the netstat output: >> Active Internet connections (w/o servers) >> Proto Recv-Q Send-Q Local Address Foreign Address State >> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 >> ESTABLISHED >> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 >> ESTABLISHED >> tcp 0 0 localhost.loca:postgres localhost.localdo:36846 >> ESTABLISHED > > That's not helpful. I want to see what the JVM process is holding, not what > the whole OS is handling. > > My guess is that lsof is going to be a better bet for inspecting the process. > But if you get "too many open files" being thrown inside the JVM and you > don't see a 1/4 million lines of "lsof" output on your process, then > something else is going on. But remember to limit your lsof to *just the JVM* > process. > > -chris > >> -----Original Message----- >> From: Christopher Schultz <ch...@christopherschultz.net> >> Sent: Monday, May 17, 2021 5:18 PM >> To: users@tomcat.apache.org >> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 >> >> Yeggy, >> >> On 5/17/21 15:17, Yeggy Javadi wrote: >>> Hi, >>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8: >>> java 51798 root 6601u sock 0,9 0t0 3959342 >>> protocol: TCPv6 >>> >>> I was running the same application on Linux 7.6 with the same >>> apache-tomcat version without seeing this issue. I saw that RedHat >>> has removed apache-tomcat fom their Linux 8 distribution. Please help >>> with throubleshooting and indicating if there is any changes in Linux >>> 8 that can explain this >>> >>> Configuration: >>> apache-tomcat-8.5.59 >>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application >>> using Postgres and elastic search databases >>> >>> >>> # ps -ef | grep tomcat >>> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java >>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties >>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 >>> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m >>> -Djdk.tls.ephemeralDHKeySize=2048 >>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources >>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 >>> -Dignore.endorsed.dirs= -classpath >>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar >>> -Dcatalina.base=/usr/local/apache-tomcat >>> -Dcatalina.home=/usr/local/apache-tomcat >>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp >>> org.apache.catalina.startup.Bootstrap start >>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat >>> >>> >>> # cat /proc/51798/limits >>> Limit Soft Limit Hard Limit Units >>> Max cpu time unlimited unlimited seconds >>> Max file size unlimited unlimited bytes >>> Max data size unlimited unlimited bytes >>> Max stack size 8388608 unlimited bytes >>> Max core file size unlimited unlimited bytes >>> Max resident set unlimited unlimited bytes >>> Max processes 256518 256518 >>> processes >>> Max open files 262144 262144 files >>> Max locked memory 65536 65536 bytes >>> Max address space unlimited unlimited bytes >>> Max file locks unlimited unlimited locks >>> Max pending signals 256518 256518 signals >>> Max msgqueue size 819200 819200 bytes >>> Max nice priority 0 0 >>> Max realtime priority 0 0 >>> Max realtime timeout unlimited unlimited us >>> >>> >>> Too may open files: >>> # lsof -p 51798 | wc -l >>> 6679 >>> >>> >>> Most of open files are TCP sockets: >>> java 51798 root 6601u sock 0,9 0t0 3959342 >>> protocol: TCPv6 >>> >>> >>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l >>> 6413 >> >> Some potentially dumb questions: >> >> 1. Did you upgrade anything recently (like Java VM)? >> >> 2. What is error you are seeing? A full stack trace would be helpful. >> >> 3. What is your <Connector> configuration? >> >> 4. Are you counting all the open files for a single process or all of the >> sub-processes which represent the threads of the main process? >> Different kernel versions count things differently. >> >> 5. Running lsof, netstat, etc. can you see if some large number of those >> sockets are bound to any specific port (listen or connect)? >> >> -chris >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org >
