ss has all the same options as netstat On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) <a...@ice-sa.com> wrote:
> Soyrry to top-post, but it's getting cluttered down there.. > > The next thing that you may want to do : > > > netstat -p -a -6 --tcp > > That is an alternative list of sockets, which also shows the "tcp state" > of the sockets. > To get only the ones of the tomcat JVM PID, filter with grep based of the > last column. > The type of thing you are looking for is the column which should show > "LISTEN", or > "ESTABLISHED" or "CLOSE_WAIT" etc.. > > The options above : > -p : show PID and program > -a : show all sockets states > -6 : only inet v6 > --tcp : only TCP sockets > > "netstat" may not be on your system by default, and you may need to > install it. > An alternative is "ss", but I don't know the options. > > > On 21.05.2021 02:14, Yeggy Javadi wrote: > > Hi, > > Yes; that is what I get and as you can see among 8028 open files, 7474 > are for TCPv6 sockets: > > java 130244 root 7805u sock 0,9 0t0 12294251 > protocol: TCPv6 > > > > # ps -ef | grep tomcat > > root 130244 1 1 11:01 ? 00:06:20 > /usr/local/jre/bin/java > -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 > -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dignore.endorsed.dirs= -classpath > /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar > -Dcatalina.base=/usr/local/apache-tomcat > -Dcatalina.home=/usr/local/apache-tomcat > -Djava.io.tmpdir=/usr/local/apache-tomcat/temp > org.apache.catalina.startup.Bootstrap start > > root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto > tomcat > > > > #lsof -p 130244 | wc -l > > 8028 > > > > #lsof -p 130244 | grep "protocol: TCPv6"| wc -l > > 7474 > > > > Thanks > > > > -----Original Message----- > > From: André Warnier (tomcat/perl) <a...@ice-sa.com> > > Sent: Thursday, May 20, 2021 4:19 PM > > To: users@tomcat.apache.org > > Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 > > > > Hi. > > According to the list below, you have 2 java (JVM) processes running on > your system. > > One (PID = 130244) is the JVM which runs tomcat. This is visible when > you look at the whole command-line. > > > > The other (PID = 130516) runs ElasticSearch, which I believe is not > relevant here. > > > > So you should run lsof with the "-p 130244" option, to show only the > files opened by the tomcat JVM. > > To show the current Tomcat JVM PID, do e.g. : ps -ef | grep > "apache-tomcat" > > (or anything else unique in that line) > > > > > > On 20.05.2021 21:00, Yeggy Javadi wrote: > >> Hi Chris, > >> Please indicate how to show lsof or netstat to *just the JVM* process. > >> > >> Below is the list of running processes on my server: > >> UID PID PPID C STIME TTY TIME CMD > >> root 1 0 0 May07 ? 00:00:14 > /usr/lib/systemd/systemd --switched-root --system --deserialize 17 > >> root 2 0 0 May07 ? 00:00:00 [kthreadd] > >> root 3 2 0 May07 ? 00:00:00 [rcu_gp] > >> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp] > >> root 6 2 0 May07 ? 00:00:00 > [kworker/0:0H-kblockd] > >> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq] > >> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0] > >> root 10 2 0 May07 ? 00:02:18 [rcu_sched] > >> root 11 2 0 May07 ? 00:00:00 [migration/0] > >> root 12 2 0 May07 ? 00:00:00 [watchdog/0] > >> root 13 2 0 May07 ? 00:00:00 [cpuhp/0] > >> root 14 2 0 May07 ? 00:00:00 [cpuhp/1] > >> root 15 2 0 May07 ? 00:00:00 [watchdog/1] > >> root 16 2 0 May07 ? 00:00:00 [migration/1] > >> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1] > >> root 19 2 0 May07 ? 00:00:00 > [kworker/1:0H-kblockd] > >> root 20 2 0 May07 ? 00:00:00 [cpuhp/2] > >> root 21 2 0 May07 ? 00:00:00 [watchdog/2] > >> root 22 2 0 May07 ? 00:00:00 [migration/2] > >> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2] > >> root 25 2 0 May07 ? 00:00:00 > [kworker/2:0H-kblockd] > >> root 26 2 0 May07 ? 00:00:00 [cpuhp/3] > >> root 27 2 0 May07 ? 00:00:00 [watchdog/3] > >> root 28 2 0 May07 ? 00:00:00 [migration/3] > >> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3] > >> root 31 2 0 May07 ? 00:00:00 > [kworker/3:0H-kblockd] > >> root 32 2 0 May07 ? 00:00:00 [cpuhp/4] > >> root 33 2 0 May07 ? 00:00:00 [watchdog/4] > >> root 34 2 0 May07 ? 00:00:00 [migration/4] > >> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4] > >> root 37 2 0 May07 ? 00:00:00 > [kworker/4:0H-kblockd] > >> root 38 2 0 May07 ? 00:00:00 [cpuhp/5] > >> root 39 2 0 May07 ? 00:00:00 [watchdog/5] > >> root 40 2 0 May07 ? 00:00:00 [migration/5] > >> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5] > >> root 43 2 0 May07 ? 00:00:00 > [kworker/5:0H-kblockd] > >> root 44 2 0 May07 ? 00:00:00 [cpuhp/6] > >> root 45 2 0 May07 ? 00:00:00 [watchdog/6] > >> root 46 2 0 May07 ? 00:00:00 [migration/6] > >> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6] > >> root 49 2 0 May07 ? 00:00:00 > [kworker/6:0H-kblockd] > >> root 50 2 0 May07 ? 00:00:00 [cpuhp/7] > >> root 51 2 0 May07 ? 00:00:00 [watchdog/7] > >> root 52 2 0 May07 ? 00:00:00 [migration/7] > >> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7] > >> root 55 2 0 May07 ? 00:00:00 > [kworker/7:0H-kblockd] > >> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs] > >> root 58 2 0 May07 ? 00:00:00 [netns] > >> root 59 2 0 May07 ? 00:00:00 [kauditd] > >> root 62 2 0 May07 ? 00:00:00 [khungtaskd] > >> root 63 2 0 May07 ? 00:00:00 [oom_reaper] > >> root 64 2 0 May07 ? 00:00:00 [writeback] > >> root 65 2 0 May07 ? 00:00:00 [kcompactd0] > >> root 66 2 0 May07 ? 00:00:00 [ksmd] > >> root 67 2 0 May07 ? 00:00:02 [khugepaged] > >> root 68 2 0 May07 ? 00:00:00 [crypto] > >> root 69 2 0 May07 ? 00:00:00 [kintegrityd] > >> root 70 2 0 May07 ? 00:00:00 [kblockd] > >> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq] > >> root 72 2 0 May07 ? 00:00:00 [md] > >> root 73 2 0 May07 ? 00:00:00 [edac-poller] > >> root 74 2 0 May07 ? 00:00:00 [watchdogd] > >> root 92 2 0 May07 ? 00:00:00 [kswapd0] > >> root 188 2 0 May07 ? 00:00:00 [kthrotld] > >> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp] > >> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp] > >> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp] > >> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp] > >> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp] > >> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp] > >> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp] > >> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp] > >> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp] > >> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp] > >> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp] > >> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp] > >> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp] > >> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp] > >> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp] > >> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp] > >> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp] > >> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp] > >> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp] > >> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp] > >> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp] > >> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp] > >> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp] > >> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp] > >> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp] > >> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp] > >> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp] > >> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp] > >> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp] > >> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp] > >> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp] > >> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp] > >> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm] > >> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd] > >> root 224 2 0 May07 ? 00:00:00 [kaluad] > >> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf] > >> root 227 2 0 May07 ? 00:00:00 [kstrp] > >> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0] > >> root 509 2 0 May07 ? 00:00:00 [ata_sff] > >> root 510 2 0 May07 ? 00:00:00 [mpt/0] > >> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0] > >> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0] > >> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1] > >> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1] > >> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx] > >> root 525 2 0 May07 ? 00:00:00 [ttm_swap] > >> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2] > >> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2] > >> root 540 2 0 May07 ? 00:00:01 > [kworker/7:1H-kblockd] > >> root 541 2 0 May07 ? 00:00:01 > [kworker/5:1H-kblockd] > >> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8] > >> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > >> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8] > >> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > >> root 613 2 0 May07 ? 00:00:01 > [kworker/6:1H-kblockd] > >> root 650 2 0 May07 ? 00:00:01 > [kworker/4:1H-kblockd] > >> root 659 2 0 May07 ? 00:00:02 > [kworker/3:1H-kblockd] > >> root 660 2 0 May07 ? 00:00:02 > [kworker/2:1H-kblockd] > >> root 662 2 0 May07 ? 00:00:01 > [kworker/0:1H-kblockd] > >> root 679 1 0 May07 ? 00:00:03 > /usr/lib/systemd/systemd-journald > >> root 682 2 0 May07 ? 00:00:02 > [kworker/1:1H-kblockd] > >> root 716 1 0 May07 ? 00:00:01 > /usr/lib/systemd/systemd-udevd > >> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8] > >> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > >> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8] > >> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > >> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8] > >> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > >> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8] > >> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver] > >> root 776 1 0 May07 ? 00:00:01 /sbin/auditd > >> root 790 2 0 May07 ? 00:00:00 [rpciod] > >> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0] > >> root 792 2 0 May07 ? 00:00:00 [xprtiod] > >> dbus 883 1 0 May07 ? 00:00:03 > /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile > --systemd-activation --syslog-only > >> root 885 1 0 May07 ? 00:02:42 > /usr/sbin/NetworkManager --no-daemon > >> root 887 1 0 May07 ? 00:00:44 > /usr/sbin/irqbalance --foreground > >> root 888 1 0 May07 ? 00:00:00 > /usr/bin/VGAuthService -s > >> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd > >> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i > --logger=files > >> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd > >> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f > --fill-watermark=0 > >> root 946 890 0 May07 ? 00:00:06 > /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 > --logger=files > >> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f > -d2 > >> root 962 890 0 May07 ? 00:00:12 > /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files > >> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy > -D > >> root 970 1 0 May07 ? 00:00:02 > /usr/lib/systemd/systemd-logind > >> root 971 2 0 May07 ? 00:00:00 [iscsi_eh] > >> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd > -n > >> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n > >> root 6223 1 0 May07 ? 00:07:27 > /usr/local/vtl/bin/sanidmgr > >> root 6271 2 0 May07 ? 00:00:03 [bksc_exec] > >> root 6272 2 0 May07 ? 00:00:03 [bksc_done] > >> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3] > >> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3] > >> root 6460 2 0 May07 ? 00:00:00 [kdownstream] > >> root 6461 2 0 May07 ? 00:00:07 [msgfilter] > >> root 6463 2 0 May07 ? 00:00:12 [iocore] > >> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr] > >> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr] > >> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0 > >> root 6523 2 0 May07 ? 00:01:50 [trudpd] > >> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker] > >> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check] > >> root 6551 2 0 May07 ? 00:00:00 [misc_thread] > >> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen] > >> root 6554 2 0 May07 ? 00:00:00 [listen_handle] > >> root 6557 2 0 May07 ? 00:00:00 [ipstord] > >> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq] > >> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4] > >> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4] > >> root 6654 1 0 May07 ? 00:00:00 > /usr/local/vtl/bin/ipstorcomm -d 0 > >> root 6655 6654 0 May07 ? 00:05:18 > /usr/local/vtl/bin/ipstorcomm -d 0 > >> root 6656 6655 0 May07 ? 00:00:36 > /usr/local/vtl/bin/ipstorcomm -d 0 > >> root 6693 1 0 May07 ? 00:00:00 rdesirkey > >> root 6699 1 0 May07 ? 00:00:12 ikmd > >> root 6721 1 0 May07 ? 00:00:35 > /usr/local/vtl/bin/iscliproxy > >> root 6747 1 0 May07 ? 00:00:40 > /usr/local/vtl/bin/ipstorlog > >> root 6780 1 0 May07 ? 00:04:50 > /usr/local/vtl/bin/snmpd > >> root 7722 1 0 May07 ? 00:00:00 > /usr/local/vtl/bin/ipstorsm > >> root 7723 7722 0 May07 ? 01:08:12 > /usr/local/vtl/bin/ipstorsm > >> root 7901 1 0 May07 ? 00:00:00 > /usr/lib/systemd/systemd --user > >> root 7904 7901 0 May07 ? 00:00:00 (sd-pam) > >> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p > -- \u --noclear tty1 linux > >> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv] > >> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0 > >> root 14202 14201 0 May08 pts/0 00:00:00 -bash > >> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D > -oCiphers=aes256-...@openssh.com,chacha20-poly1...@openssh.com > ,aes256-ctr,aes256-cbc,aes128-...@openssh.com,aes128-ctr,aes128-cbc > -oMACs=hmac-sha2-256-...@openssh.com,hmac-sha1-...@openssh.com, > umac-128-...@openssh.com,hmac-sha2-512-...@openssh.com > ,hmac-sha2-256,hmac-sha1,umac-...@openssh.com,hmac-sha2-512 > -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1- > -oKexAlgorithms=curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 > -oHostKeyAlgorithms=ecdsa-sha2-nistp256, > ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384, > ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521, > ecdsa-sha2-nistp521-cert-...@openssh.com,ssh-ed25519, > ssh-ed25519-cert-...@openssh.com,rsa-sha2-256, > rsa-sha2-256-cert-...@openssh.com,rsa-sha2-512, > rsa-sha2-512-cert-...@openssh.com,ssh-rsa,ssh-rsa-cert-...@openssh.com > -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256, > ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384, > ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521, > ecdsa-sha2-nistp521-cert-...@openssh.com,ssh-ed25519, > ssh-ed25519-cert-...@openssh.com,rsa-sha2-256, > rsa-sha2-256-cert-...@openssh.com,rsa-sha2-512, > rsa-sha2-512-cert-...@openssh.com,ssh-rsa,ssh-rsa-cert-...@openssh.com > -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa > >> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events] > >> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events] > >> root 128108 2 0 08:40 ? 00:00:00 > [kworker/4:1-events_power_efficient] > >> root 128360 2 0 09:10 ? 00:00:00 > [kworker/4:2-cgroup_destroy] > >> root 128430 2 0 09:30 ? 00:00:00 > [kworker/6:0-mm_percpu_wq] > >> root 128738 2 0 11:00 ? 00:00:00 > [kworker/3:3-mm_percpu_wq] > >> root 128761 2 0 11:00 ? 00:00:00 > [kworker/2:1-mm_percpu_wq] > >> root 130244 1 1 11:01 ? 00:03:50 > /usr/local/jre/bin/java > -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 > -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dignore.endorsed.dirs= -classpath > /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar > -Dcatalina.base=/usr/local/apache-tomcat > -Dcatalina.home=/usr/local/apache-tomcat > -Djava.io.tmpdir=/usr/local/apache-tomcat/temp > org.apache.catalina.startup.Bootstrap start > >> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g > -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC > -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes > -Des.path.home=/usr/share/elasticsearch -cp > :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* > -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid > -Des.default.path.home=/usr/share/elasticsearch > -Des.default.path.logs=/var/log/elasticsearch > -Des.default.path.data=/var/lib/elasticsearch > -Des.default.config=/etc/elasticsearch/elasticsearch.yml > -Des.default.path.conf=/etc/elasticsearch > org.elasticsearch.bootstrap.Elasticsearch > >> postgres 130548 1 0 11:01 ? 00:00:00 > /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data > >> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger > process > >> postgres 130563 130548 0 11:01 ? 00:00:00 postgres: > checkpointer process > >> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer > process > >> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal > writer process > >> postgres 130566 130548 0 11:01 ? 00:00:00 postgres: > autovacuum launcher process > >> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats > collector process > >> root 130570 1 0 11:01 ? 00:00:01 > /usr/local/freestor/bin/fmsd > >> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres > freestor 127.0.0.1(42660) idle > >> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres > freestor 127.0.0.1(42662) idle > >> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres > freestor 127.0.0.1(42664) idle > >> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2] > >> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2] > >> root 131347 2 0 13:49 ? 00:00:00 > [kworker/5:1-mm_percpu_wq] > >> root 131364 2 0 13:56 ? 00:00:00 > [kworker/u16:2-events_unbound] > >> root 131401 2 0 14:05 ? 00:00:00 > [kworker/u16:1-events_unbound] > >> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2] > >> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events] > >> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0] > >> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events] > >> root 131515 2 0 14:49 ? 00:00:00 > [kworker/1:0-events_freezable] > >> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0] > >> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv] > >> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1 > >> root 131530 131529 0 14:52 pts/1 00:00:00 -bash > >> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef > >> > >> Thanks > >> -----Original Message----- > >> From: Christopher Schultz <ch...@christopherschultz.net> > >> Sent: Thursday, May 20, 2021 12:49 PM > >> To: users@tomcat.apache.org > >> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 > >> > >> Yeggy, > >> > >> On 5/19/21 19:37, Yeggy Javadi wrote: > >>> Hi Chris, > >>> Thanks for your email. Below are replies to your questions: > >>> > >>> 1. Did you upgrade anything recently (like Java VM)? > >>> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 > to 9.6. > >>> > >>> 2. What is error you are seeing? A full stack trace would be helpful. > >>> [YJ] The application error can occur in any place with the (Too many > open files) error when the limit of open files for tomcat has reached > (262144 in my environment), for example, I can get a connectivity error > when pulling info from a server as below : > >>> > >>> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - > >>> Error : The XML configuration file failed to be retrieved for server > >>> 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for > >>> server=10.1.6.25 error=java.io.FileNotFoundException: > >>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3. > >>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output= > >>> > >>> 3. What is your <Connector> configuration? > >>> [YJ] It is as below in server.xml: > >>> > >>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1" > >>> maxThreads="150" > >>> minSpareThreads="25" maxSpareThreads="75" > >>> enableLookups="false" acceptCount="100" > >>> connectionTimeout="20000" disableUploadTimeout="true" > >>> compression="on" > >>> compressionMinSize="2048" > >>> noCompressionUserAgents="gozilla, traviata" > >>> > compressableMimeType="text/html,text/xml,application/json" > >>> redirectPort="443" /> > >>> ........... > >>> <Connector address="0.0.0.0" port="443" > protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true" > >>> maxThreads="150" scheme="https" secure="true" > >>> clientAuth="false" sslEnabledProtocols="TLSv1.2" > >>> > > ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA" > >>> keystoreFile="conf/freestor" > keystorePass="freestor"/> > >>> > >>> <!-- Define an AJP 1.3 Connector on port 8009 --> > >>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3" > >>> redirectPort="443" secretRequired="false" /> > >> > >> You have lots of settings on the HTTP connector but not the HTTPS one. > >> Is that intentional? Which one are you using? Or both? > >> > >> Are you using AJP? If not, disable the connetor entirely. Definitely > never enable it on 0.0.0.0 unless you really know what you are doing. > >> > >>> 4. Are you counting all the open files for a single process or all of > the sub-processes which represent the threads of the main process? > >>> Different kernel versions count things differently. > >>> [YJ] I am just getting the process ID of tomcat and count open files > >>> for that process > >> > >> Ok. > >> > >>> 5. Running lsof, netstat, etc. can you see if some large number of > those sockets are bound to any specific port (listen or connect)? > >>> [YJ] Here is the netstat output: > >>> Active Internet connections (w/o servers) > >>> Proto Recv-Q Send-Q Local Address Foreign Address > State > >>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 > ESTABLISHED > >>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 > ESTABLISHED > >>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846 > ESTABLISHED > >> > >> That's not helpful. I want to see what the JVM process is holding, not > what the whole OS is handling. > >> > >> My guess is that lsof is going to be a better bet for inspecting the > process. But if you get "too many open files" being thrown inside the JVM > and you don't see a 1/4 million lines of "lsof" output on your process, > then something else is going on. But remember to limit your lsof to *just > the JVM* process. > >> > >> -chris > >> > >>> -----Original Message----- > >>> From: Christopher Schultz <ch...@christopherschultz.net> > >>> Sent: Monday, May 17, 2021 5:18 PM > >>> To: users@tomcat.apache.org > >>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 > >>> > >>> Yeggy, > >>> > >>> On 5/17/21 15:17, Yeggy Javadi wrote: > >>>> Hi, > >>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8: > >>>> java 51798 root 6601u sock 0,9 0t0 3959342 > protocol: TCPv6 > >>>> > >>>> I was running the same application on Linux 7.6 with the same > >>>> apache-tomcat version without seeing this issue. I saw that RedHat > >>>> has removed apache-tomcat fom their Linux 8 distribution. Please help > >>>> with throubleshooting and indicating if there is any changes in Linux > >>>> 8 that can explain this > >>>> > >>>> Configuration: > >>>> apache-tomcat-8.5.59 > >>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application > >>>> using Postgres and elastic search databases > >>>> > >>>> > >>>> # ps -ef | grep tomcat > >>>> root 51798 1 1 11:16 ? 00:06:01 > /usr/local/jre/bin/java > -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 > -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dignore.endorsed.dirs= -classpath > /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar > -Dcatalina.base=/usr/local/apache-tomcat > -Dcatalina.home=/usr/local/apache-tomcat > -Djava.io.tmpdir=/usr/local/apache-tomcat/temp > org.apache.catalina.startup.Bootstrap start > >>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto > tomcat > >>>> > >>>> > >>>> # cat /proc/51798/limits > >>>> Limit Soft Limit Hard Limit > Units > >>>> Max cpu time unlimited unlimited > seconds > >>>> Max file size unlimited unlimited > bytes > >>>> Max data size unlimited unlimited > bytes > >>>> Max stack size 8388608 unlimited > bytes > >>>> Max core file size unlimited unlimited > bytes > >>>> Max resident set unlimited unlimited > bytes > >>>> Max processes 256518 256518 > processes > >>>> Max open files 262144 262144 > files > >>>> Max locked memory 65536 65536 > bytes > >>>> Max address space unlimited unlimited > bytes > >>>> Max file locks unlimited unlimited > locks > >>>> Max pending signals 256518 256518 > signals > >>>> Max msgqueue size 819200 819200 > bytes > >>>> Max nice priority 0 0 > >>>> Max realtime priority 0 0 > >>>> Max realtime timeout unlimited unlimited us > >>>> > >>>> > >>>> Too may open files: > >>>> # lsof -p 51798 | wc -l > >>>> 6679 > >>>> > >>>> > >>>> Most of open files are TCP sockets: > >>>> java 51798 root 6601u sock 0,9 0t0 3959342 > protocol: TCPv6 > >>>> > >>>> > >>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l > >>>> 6413 > >>> > >>> Some potentially dumb questions: > >>> > >>> 1. Did you upgrade anything recently (like Java VM)? > >>> > >>> 2. What is error you are seeing? A full stack trace would be helpful. > >>> > >>> 3. What is your <Connector> configuration? > >>> > >>> 4. Are you counting all the open files for a single process or all of > the sub-processes which represent the threads of the main process? > >>> Different kernel versions count things differently. > >>> > >>> 5. Running lsof, netstat, etc. can you see if some large number of > those sockets are bound to any specific port (listen or connect)? > >>> > >>> -chris > >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554
