Hello,
I have an application running under tomcat configured with
Http11NioProtocol. I am noticing that under a high connection rate (not
request rate) of about 100 connections/second, tomcat is leaks file
handles corresponding to Unix sockets. I am using the lsof command to find
the number of file handles open by the java process. I observe an increase
number of file handles that are all pointing to the same unix socket (same
inode number) over a period of time.
# ls -l /proc/3907/fd/8368
lrwx------ 1 root root 64 Jan 7 00:15 /proc/3907/fd/8368 ->
socket:[605702067]
# ls -l /proc/3907/fd/8366
lrwx------ 1 root root 64 Jan 7 00:15 /proc/3907/fd/8366 ->
socket:[605702067]
# ls -l /proc/3907/fd/8367
lrwx------ 1 root root 64 Jan 7 00:15 /proc/3907/fd/8367 ->
socket:[605702067]
Netstat –p does show that the state of the socket is connected.
# netstat -p | grep unix | grep java
unix 2 [ ] STREAM CONNECTED 605702067 3907/java
At my anticipated connection rates (100/second), we observe the application
to leak about 150 file handles (corresponding to Unix sockets) each day. I
did some investigation on the issue by connecting to the application via
JMX. It seems like this increase in Unix socket file handles is due to
improperly closed connections, since I am also able to reproduce the same
unix socket leak by invoking stop() on the HTTP Connector via JMX. Just
before stopping the connector my application had 4000 open connections
(lsof showed these as 4000 IPv6 file handles). On invoking stop() via JMX,
I the number of IPv6 file went down to about 70, while the number of unix
socket file handles went up by 4000. It seems that these incorrectly closed
connections then incarnate as Unix socket handles.
Here;s the distribution of file handles before and after the stop():
Command used: pid=`ps -ef | grep tomcat | grep -v grep | awk '{print
$2}'`; lsof -p $pid | awk '{print $5}' | sort -n | uniq -c
Before
10 0000
4 CHR
2 DIR
20 FIFO
4105 IPv6 -- see the drop
93 REG
37 sock
1 TYPE
12 unix
1 unknown
After:
Sat Jan 7 00:07:50 SAST 2012
10 0000
4 CHR
2 DIR
20 FIFO
13 IPv6
93 REG
1 sock
1 TYPE
4138 unix -- see the increase
A similar issue about leaking unix sockets is also reported in
http://stackoverflow.com/questions/7038688/java-nio-causes-file-descriptor-leak.
The same post also mentions that NIO internally uses Unix sockets, and the
problem was with the way the application was using NIO. I suspect there is
a race condition in the NIO handler that causes some resources not getting
cleaned up properly under high connection rate.
I have tried different versions of tomcat 6.0.32, 6.0.35, 7.0.21 and
tomcat7.0.23 – they all have this issue. My application is using the
default configuration of the NIO Connector. I dont think Comet or SendFile
are being used. Furthermore, no unix sockets show up when I use Blocking IO
(default HTTP handler).
I would appreciate the help of the community in addressing this issue.
Thanks,
Arijit
