The network card of our DB server often freezes and all applications running
on APP are therefore failing. DB hard reboot necessary.
Let me first introduce the configuration:
Dell Poweredge 2400 Application server (APP) connected to Dell Poweredge
2400 Database server (DB)
Both are running under Windows NT SP6a
DB has Oracle 8.1.6
APP has Apache 2.0.52 with mod_jk 1.2.13, Tomcat 5.5.9 and JDK 1_5_0_04
(APP was recently migrated from IBM HTTP Server 2 and Websphere Application
Server 4.0)
DB was running stable for a couple of years.
APP is running 6 applications sharing 3 DBCP connection pooling (masked
connection string):
<Resource name="db/local" scope="Shareable" type="javax.sql.DataSource"
auth="Container"
url="jdbc:oracle:thin:@file_db:1521:DB" password="pass" maxActive="30"
maxWait="-1"
driverClassName="oracle.jdbc.driver.OracleDriver" username="xxx"
maxIdle="10" />
<Resource name="dbnl/local" scope="Shareable"
type="javax.sql.DataSource" auth="Container"
url="jdbc:oracle:thin:@file_db:1521:DB" password="pass2" maxActive="20"
maxWait="-1"
driverClassName="oracle.jdbc.driver.OracleDriver" username="xxx2"
maxIdle="10" />
<Resource name="dbuk/local" scope="Shareable"
type="javax.sql.DataSource" auth="Container"
url="jdbc:oracle:thin:@file_db:1521:DB" password="pass3" maxActive="30"
maxWait="-1"
driverClassName="oracle.jdbc.driver.OracleDriver" username="xxx3"
maxIdle="10" />
<Connector port="8009"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="8443" protocol="AJP/1.3"
/>
Under certain conditions (seems to be heavier load) the network card of the
DB freezes and all applications running on APP are therefore failing. I've
checked all Oracle logs and Oracle is not crashing. The number of DB
connections is even low (around 30 while 150 permitted) and no new
connections are being created just before the crash.
The Apache 2 server status is not showing an abnormal high number of
connections either leading me to think Apache HTTP and mod_jk are not at
fault.
I am suspecting a race condition with DBCP connection pooling on Tomcat
5.5.9 which increases network traffic causing the network card of the DB
server to crash. Our DB server currently crashes 2 times a week.
I am planning to try installing tomcat 5.5.7 instead as I've seen some
changes logged in 5.5.9 regarding the ms thread pool (but at this stage I am
really guessing)
Problem also occurs using the AbandonnedPool option of Commons DBCP, ie with
following extra parameters in the Resource tag
removeAbandoned="true" removeAbandonedTimeout="300" logAbandoned="true"
stdout then contains the only visible error but error might be logical
since network is also interrupted.
Please note that the connections are well closed programmatically as those
functions are called regularly.
It is just that when (before or after) DB crashes connections don't seem to
close.
All help/suggestions are welcome.
Thanks
StdOut:
-------
DBCP object created 2005-08-03 17:42:47 by the following code was never
closed:
java.lang.Exception
at
org.apache.tomcat.dbcp.dbcp.AbandonedTrace.setStackTrace(AbandonedTrace.java
:157)
at
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObject
Pool.java:76)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSourc
e.java:95)
at
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.ja
va:540)
at com.bobex.dao.BaseDAO.getConnection(BaseDAO.java:176)
at com.bobex.dao.BidDAO.findFeedbackOfBids(BidDAO.java:742)
at
com.bobex.servlets.events.BidFeedbackListEventListener.requery(BidFeedbackLi
stEventListener.java:414)
at
com.bobex.util.list.events.AbstractListModel.fireNext(AbstractListModel.java
:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
com.bobex.util.list.events.AbstractListModel.listEventOccured(AbstractListMo
del.java:685)
at com.bobex.taglibs.list.ListTag.doStartTag(ListTag.java:77)
at
org.apache.jsp.mybobex.feedback_jsp._jspx_meth_util_list_0(org.apache.jsp.my
bobex.feedback_jsp:141)
at
org.apache.jsp.mybobex.feedback_jsp._jspService(org.apache.jsp.mybobex.feedb
ack_jsp:103)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
22)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:291)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:173)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
java:672)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
patcher.java:463)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
er.java:398)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
.java:301)
at com.bobex.servlets.HttpServletState.forward(HttpServletState.java:49)
at
com.bobex.servlets.BaseControllerServlet.forwardToNextAction(BaseControllerS
ervlet.java:70)
at
com.bobex.servlets.MyBobexAreaServlet.viewFeedbackOfBids(MyBobexAreaServlet.
java:1371)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
com.bobex.servlets.MethodCallerServlet.handleRequest(MethodCallerServlet.jav
a:47)
at
com.bobex.servlets.BaseControllerServlet.service(BaseControllerServlet.java:
175)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:173)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
java:672)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
patcher.java:463)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
er.java:398)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
.java:301)
at com.bobex.servlets.HttpServletState.forward(HttpServletState.java:49)
at
com.bobex.servlets.DispatchServlet.handleRequest(DispatchServlet.java:139)
at
com.bobex.servlets.BaseControllerServlet.service(BaseControllerServlet.java:
175)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)
at java.lang.Thread.run(Unknown Source)
DBCP object created 2005-08-03 17:42:47 by the following code was never
closed:
java.lang.Exception
at org.apache.tomcat.dbcp.dbcp.AbandonedTrace.init(AbandonedTrace.java:96)
at
org.apache.tomcat.dbcp.dbcp.AbandonedTrace.<init>(AbandonedTrace.java:79)
at
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.<init>(DelegatingStatement.j
ava:60)
at
org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.<init>(DelegatingPre
paredStatement.java:65)
at
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.prepareStatement(Delegating
Connection.java:185)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.pre
pareStatement(PoolingDataSource.java:278)
at com.bobex.dao.BidDAO.findFeedbackOfBids(BidDAO.java:743)
at
com.bobex.servlets.events.BidFeedbackListEventListener.requery(BidFeedbackLi
stEventListener.java:414)
at
com.bobex.util.list.events.AbstractListModel.fireNext(AbstractListModel.java
:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
com.bobex.util.list.events.AbstractListModel.listEventOccured(AbstractListMo
del.java:685)
at com.bobex.taglibs.list.ListTag.doStartTag(ListTag.java:77)
at
org.apache.jsp.mybobex.feedback_jsp._jspx_meth_util_list_0(org.apache.jsp.my
bobex.feedback_jsp:141)
at
org.apache.jsp.mybobex.feedback_jsp._jspService(org.apache.jsp.mybobex.feedb
ack_jsp:103)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
22)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:291)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:173)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
java:672)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
patcher.java:463)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
er.java:398)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
.java:301)
at com.bobex.servlets.HttpServletState.forward(HttpServletState.java:49)
at
com.bobex.servlets.BaseControllerServlet.forwardToNextAction(BaseControllerS
ervlet.java:70)
at
com.bobex.servlets.MyBobexAreaServlet.viewFeedbackOfBids(MyBobexAreaServlet.
java:1371)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
com.bobex.servlets.MethodCallerServlet.handleRequest(MethodCallerServlet.jav
a:47)
at
com.bobex.servlets.BaseControllerServlet.service(BaseControllerServlet.java:
175)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:173)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
java:672)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
patcher.java:463)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
er.java:398)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
.java:301)
at com.bobex.servlets.HttpServletState.forward(HttpServletState.java:49)
at
com.bobex.servlets.DispatchServlet.handleRequest(DispatchServlet.java:139)
at
com.bobex.servlets.BaseControllerServlet.service(BaseControllerServlet.java:
175)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)
at java.lang.Thread.run(Unknown Source)
DBCP object created 2005-08-03 17:43:04 by the following code was never
closed:
java.lang.Exception
at
org.apache.tomcat.dbcp.dbcp.AbandonedTrace.setStackTrace(AbandonedTrace.java
:157)
at
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObject
Pool.java:76)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSourc
e.java:95)
at
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.ja
va:540)
at com.bobex.dao.BaseDAO.getConnection(BaseDAO.java:176)
at com.bobex.dao.UserDAO.findUserByUsername(UserDAO.java:1357)
at com.bobex.beans.LoginBean.login(LoginBean.java:103)
at com.bobex.servlets.LoginServlet.login(LoginServlet.java:159)
at sun.reflect.GeneratedMethodAccessor1866.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
... similar errors follow almost at same time of crash
David
