Hi Chris,
Thanks for your email. Below are replies to your questions:
1. Did you upgrade anything recently (like Java VM)?
[YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
2. What is error you are seeing? A full stack trace would be helpful.
[YJ] The application error can occur in any place with the (Too many open
files) error when the limit of open files for tomcat has reached (262144 in my
environment), for example, I can get a connectivity error when pulling info
from a server as below :
[InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - Error : The
XML configuration file failed to be retrieved for server 10.1.6.25. Check
server connectivity.Error retrieving IPStorConfg for server=10.1.6.25
error=java.io.FileNotFoundException:
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.2.10.RELEASE.jar
(Too many open files) restatus=-1 output=
3. What is your <Connector> configuration?
[YJ] It is as below in server.xml:
<Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
maxThreads="150"
minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml,application/json"
redirectPort="443" />
...........
<Connector address="0.0.0.0" port="443"
protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslEnabledProtocols="TLSv1.2"
ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
keystoreFile="conf/freestor" keystorePass="freestor"/>
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
redirectPort="443" secretRequired="false" />
4. Are you counting all the open files for a single process or all of the
sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
[YJ] I am just getting the process ID of tomcat and count open files for that
process
5. Running lsof, netstat, etc. can you see if some large number of those
sockets are bound to any specific port (listen or connect)?
[YJ] Here is the netstat output:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36850 ESTABLISHED
tcp 0 0 localhost.localdo:11753 localhost.localdo:51302 ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36844 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48922 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48964 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48968 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60817 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48968 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48974 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60771 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48934 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48936 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48954 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48970 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48932 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48938 ESTABLISHED
tcp6 0 0 localhost.localdo:51302 localhost.localdo:11753 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48956 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48928 ESTABLISHED
tcp6 0 0 localhost.localdo:36844 localhost.loca:postgres ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48930 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 localhost.localdo:35202 localhost.localdo:vrace ESTABLISHED
tcp6 0 0 localhost.localdo:36850 localhost.loca:postgres ESTABLISHED
tcp6 0 0 localhost.localdo:vrace localhost.localdo:35202 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48966 ESTABLISHED
tcp6 0 0 localhost.localdo:51298 localhost.localdo:11753 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48954 ESTABLISHED
tcp6 0 1045 Yeggy-F8-FMSVA:54246 172.22.22.192:https ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48970 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48918 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48972 ESTABLISHED
tcp6 0 0 localhost.localdo:36846 localhost.loca:postgres ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48960 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48960 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48974 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60714 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48924 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48924 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48924 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48972 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48926 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48920 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60772 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48976 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60864 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48916 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48920 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60818 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48976 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48964 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:36321 Yeggy-F8-FMSVA:35998 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48966 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60865 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48952 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60713 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48934 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48958 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48916 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48936 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48962 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48938 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:35998 Yeggy-F8-FMSVA:36321 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48918 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48962 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48926 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48922 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48928 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48930 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48952 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48940 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48958 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48956 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48940 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48932 ESTABLISHED
udp 0 0 localhost.localdo:41912 localhost.localdo:41912 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ] DGRAM 520
/var/run/chrony/chronyd.sock
unix 3 [ ] DGRAM 14708 /run/systemd/notify
unix 2 [ ] DGRAM 14710
/run/systemd/cgroups-agent
unix 12 [ ] DGRAM 14720
/run/systemd/journal/dev-log
unix 6 [ ] DGRAM 14734
/run/systemd/journal/socket
unix 2 [ ] DGRAM 31223
/run/user/0/systemd/notify
unix 2 [ ] STREAM CONNECTED 11157518
unix 2 [ ] STREAM CONNECTED 11152644
unix 3 [ ] STREAM CONNECTED 11657
unix 3 [ ] STREAM CONNECTED 22048
/var/lib/sss/pipes/private/sbus-monitor
unix 3 [ ] STREAM CONNECTED 545
/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 11974
/var/lib/sss/pipes/private/sbus-dp_implicit_files.946
unix 2 [ ] STREAM CONNECTED 11367562
unix 3 [ ] DGRAM 31226
unix 3 [ ] STREAM CONNECTED 22051
/var/lib/sss/pipes/private/sbus-dp_implicit_files.946
unix 3 [ ] STREAM CONNECTED 31229
unix 2 [ ] DGRAM 551
unix 3 [ ] STREAM CONNECTED 23869
/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 11879
unix 2 [ ] STREAM CONNECTED 800347
unix 2 [ ] DGRAM 23301
unix 3 [ ] STREAM CONNECTED 11819
unix 2 [ ] DGRAM 246
unix 3 [ ] STREAM CONNECTED 543
unix 3 [ ] STREAM CONNECTED 23109
/run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 11365631
unix 2 [ ] DGRAM 31213
unix 3 [ ] STREAM CONNECTED 11363838
unix 3 [ ] STREAM CONNECTED 12063
unix 3 [ ] STREAM CONNECTED 797101
unix 2 [ ] DGRAM 11659
unix 3 [ ] DGRAM 14712
unix 3 [ ] STREAM CONNECTED 23548
unix 3 [ ] STREAM CONNECTED 11880
/run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 23325
unix 3 [ ] STREAM CONNECTED 248
unix 3 [ ] STREAM CONNECTED 11973
unix 2 [ ] STREAM CONNECTED 32574
unix 3 [ ] STREAM CONNECTED 507
unix 3 [ ] STREAM CONNECTED 21947
/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 24095
unix 3 [ ] STREAM CONNECTED 18281
/run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11001952
unix 2 [ ] DGRAM 11971
unix 3 [ ] STREAM CONNECTED 12134
/run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11325964
unix 3 [ ] STREAM CONNECTED 23050
unix 2 [ ] STREAM CONNECTED 11365500
unix 3 [ ] STREAM CONNECTED 24096
/run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11143790
unix 2 [ ] STREAM CONNECTED 793273
unix 3 [ ] STREAM CONNECTED 23868
unix 3 [ ] STREAM CONNECTED 11820
/run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11152638
unix 3 [ ] STREAM CONNECTED 24676
/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 23167
unix 3 [ ] STREAM CONNECTED 21912
unix 3 [ ] STREAM CONNECTED 543420
unix 3 [ ] DGRAM 31225
unix 3 [ ] STREAM CONNECTED 564
unix 3 [ ] STREAM CONNECTED 12133
unix 3 [ ] STREAM CONNECTED 12064
/var/lib/sss/pipes/private/sbus-monitor
unix 3 [ ] STREAM CONNECTED 18285
/run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 808539
unix 2 [ ] DGRAM 23542
unix 3 [ ] STREAM CONNECTED 22708
unix 2 [ ] DGRAM 11969
unix 3 [ ] STREAM CONNECTED 18279
/run/systemd/journal/stdout
unix 3 [ ] DGRAM 14713
unix 2 [ ] DGRAM 34975
unix 2 [ ] STREAM CONNECTED 11353853
unix 3 [ ] STREAM CONNECTED 11156644
/run/systemd/journal/stdout
unix 2 [ ] DGRAM 21925
unix 2 [ ] DGRAM 22033
unix 3 [ ] STREAM CONNECTED 23168
/run/systemd/journal/stdout
unix 2 [ ] DGRAM 24657
unix 3 [ ] STREAM CONNECTED 543421
unix 2 [ ] STREAM CONNECTED 793268
unix 3 [ ] STREAM CONNECTED 22047
unix 3 [ ] STREAM CONNECTED 782556
/run/systemd/journal/stdout
unix 2 [ ] DGRAM 21490
unix 3 [ ] STREAM CONNECTED 23698
unix 2 [ ] DGRAM 31210
unix 3 [ ] STREAM CONNECTED 11141980
unix 2 [ ] STREAM CONNECTED 11344541
unix 2 [ ] STREAM CONNECTED 11353876
unix 2 [ ] STREAM CONNECTED 11367557
unix 3 [ ] STREAM CONNECTED 247
unix 3 [ ] STREAM CONNECTED 544
unix 3 [ ] STREAM CONNECTED 34094
/run/dbus/system_bus_socket
unix 2 [ ] STREAM CONNECTED 25346
unix 3 [ ] DGRAM 11664
unix 3 [ ] STREAM CONNECTED 21861
unix 3 [ ] STREAM CONNECTED 13539
/run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 565
/var/lib/sss/pipes/private/sbus-monitor
unix 2 [ ] DGRAM 24744
unix 3 [ ] STREAM CONNECTED 18283
/run/systemd/journal/stdout
unix 2 [ ] DGRAM 13499
unix 2 [ ] STREAM CONNECTED 31209
unix 2 [ ] STREAM CONNECTED 32883
unix 3 [ ] DGRAM 11663
unix 3 [ ] STREAM CONNECTED 28573
/run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 561
/run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 11363839
unix 3 [ ] STREAM CONNECTED 11370377
/var/lib/sss/pipes/nss
unix 3 [ ] STREAM CONNECTED 22007
unix 2 [ ] DGRAM 28344
unix 2 [ ] DGRAM 23311
unix 3 [ ] STREAM CONNECTED 11932
unix 2 [ ] STREAM CONNECTED 11143887
unix 3 [ ] STREAM CONNECTED 31205
unix 2 [ ] DGRAM 32811
unix 2 [ ] DGRAM 513
Active Bluetooth connections (w/o servers)
Proto Destination Source State PSM DCID SCID
IMTU OMTU Security
Proto Destination Source State Channel
Thanks
-----Original Message-----
From: Christopher Schultz <[email protected]>
Sent: Monday, May 17, 2021 5:18 PM
To: [email protected]
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Yeggy,
On 5/17/21 15:17, Yeggy Javadi wrote:
> Hi,
> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
> java 51798 root 6601u sock 0,9 0t0 3959342
> protocol: TCPv6
>
> I was running the same application on Linux 7.6 with the same
> apache-tomcat version without seeing this issue. I saw that RedHat has
> removed apache-tomcat fom their Linux 8 distribution. Please help with
> throubleshooting and indicating if there is any changes in Linux 8
> that can explain this
>
> Configuration:
> apache-tomcat-8.5.59
> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
> using Postgres and elastic search databases
>
>
> # ps -ef | grep tomcat
> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
> -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>
>
> # cat /proc/51798/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size unlimited unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 256518 256518 processes
> Max open files 262144 262144 files
> Max locked memory 65536 65536 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 256518 256518 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
>
>
> Too may open files:
> # lsof -p 51798 | wc -l
> 6679
>
>
> Most of open files are TCP sockets:
> java 51798 root 6601u sock 0,9 0t0 3959342
> protocol: TCPv6
>
>
> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
> 6413
Some potentially dumb questions:
1. Did you upgrade anything recently (like Java VM)?
2. What is error you are seeing? A full stack trace would be helpful.
3. What is your <Connector> configuration?
4. Are you counting all the open files for a single process or all of the
sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
5. Running lsof, netstat, etc. can you see if some large number of those
sockets are bound to any specific port (listen or connect)?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
