Our production broker (v.5.6.0) keeps dying while in heavy use. The broker log is filled with:
2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector - Could not accept connection : java.net.SocketException: Too many open files 2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector - Could not accept connection : java.net.SocketException: Too many open files 2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector - Could not accept connection : java.net.SocketException: Too many open files This is logged at such a rapid rate that the logs roll and hide the initial error/warning. We capture open fd of the broker's process and notice that when the broker starts to croak the open fd count just explodes. Here is part of the open fd log. The first column shows broker's open fds and each line is logged every 60 secs. 1284 12569 160194 -- normal count 1294 12669 161438 1305 12779 162812 1318 12909 164426 1328 13009 165658 --------- FD explosion 1393 13659 173816 1528 15009 190748 1611 15839 201152 1701 16739 212419 1951 19239 243520 2310 22830 290374 2667 26399 332362 3013 29859 375262 3369 33422 422638 3729 37019 464017 4111 40841 515342 4484 44570 561933 4870 48432 609992 5249 52219 652157 5634 56071 705356 6019 59919 747457 6484 64571 811476 6892 68652 862375 7307 72802 914122 7727 77002 966555 8129 81022 1016717 8336 83090 1042601 8336 83090 1042584 8336 83090 1042583 It normally shows ~1300 fds and this is more or less constant overtime, but eventually it rapidly increases to 8336 and the broker becomes unusable. The ulimit is set to 4094. The netstat shows a ton of sockets in CLOSE_WAIT suggesting that the broker is not closing its side of a socket. I found related open issue https://issues.apache.org/jira/browse/AMQ-4531?page=com.atlassian.jirafisheyeplugin:fisheye-issuepanel This Jira states that the problem surfaces in 5.8.0 and when maximumConnections is set. We dont use this setting and we run with an older version of AMQ. Any ideas how to deal with this? Would closeAsync=false have any effect? JC -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-leaks-FDs-Too-many-open-files-tp4670496.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
