I've further traced the root cause of my problems with threads sitting on
socket read, never timing out, never throwing an exception and sometimes
causing application deadlock (depending on where the socket read occurs)
down to the network. By comparing the output of lsof -i :1521 | grep ....
on the webserver with the lsof -i :1521 | grep .... on the database server I
can see the webserver thinks it has more connections to the database than
the database actually has from the webserver and using the source port#
which shows up in the output of lsof on both machines I can even narrow down
to exactly what processes are missing the connections. We've made a policy
change on one of our firewalls to hopefully alleviate the problem but only
more testing will tell.
