Dear Felix, Thanks for your feedback , the health check period is so aggressive now. SRE team's and apm monitor tools call internally and externally endpoint and health indicators. I asked the sre guys to decrease the frequency of those calls. They made some decrease on calls but nothing changed. Due to an old spring mvc project it doesn't have a kubernetes probe , about 100 calls in minutes.
Gökhan On Thu, Aug 27, 2020 at 9:48 PM Felix Schumacher < felix.schumac...@internetallee.de> wrote: > > Am 27.08.20 um 11:47 schrieb Gokhan Akgul: > > Hi , > > > > I have been facing the deadlock issue for the last 2 months about > > JDBCPoolCleaner Thread . > > > > Following config set in context.xml > > > > <Resource name="jdbc/database" > > auth="Container" > > type="javax.sql.DataSource" > > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" > > driverClassName="com.mysql.jdbc.Driver" > > > url="jdbc:mysql://adress:3306/db?useUnicode=true&characterEncoding=latin5&characterResultSet=latin5&zeroDateTimeBehavior=convertToNull&autoReconnect=true&interactiveClient=true" > > username="user" > > password="pass" > > initialSize="10" > > maxActive="30" > > maxIdle="15" > > minIdle="10" > > maxWait="30000" > > timeBetweenEvictionRunsMillis="5000" > > minEvictableIdleTimeMillis="60000" > > removeAbandonedTimeout="600" > > removeAbandoned="true" > > logAbandoned="false" > > testWhileIdle="true" > > testOnBorrow="true" > > testOnReturn="false" > > validationQuery="/* ping */ SELECT 1" > > validationInterval="30000" > > jmxEnabled="true" > > > jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTimer;SlowQueryReport" > > /> > > > > > > > > Thread dump > > > > Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED > > - waiting to lock <0x57dcb0b7> (a > com.mysql.jdbc.JDBC4PreparedStatement) > > owned by http-nio-8080-exec-8 id=25 > > So, Tomcat tries to close a connection, that it thinks is abandoned > (i.e. in your setup hasn't been used for more than 600 seconds and the > pool of idle connections is getting empty). > > The connection is still in use ... > > > at > com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078) > > at > com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584) > > at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) > > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) > > at > org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388) > > at > org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468) > > at java.util.TimerThread.mainLoop(Timer.java:555) > > at java.util.TimerThread.run(Timer.java:505) > > > > Locked synchronizers: count = 1 > > - > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 > > > > > > > > http-nio-8080-exec-8 id=25 state=BLOCKED > > - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) > > owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 > > ... by hibernate. The question now is, how often is your healtheck > getting called (every 10 min == 600s)? > > If they are close together, you might want to set the abandoned timeout > higher than the healthcheck interval, or you could try to close the > connection (or whatever the equivalent is in hibernate (a session?)) in > your code. > > Felix > > > at > com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435) > > at > com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.java:3269) > > at com.mysql.jdbc.DatabaseMetaData.<init>(DatabaseMetaData.java:675) > > at > com.mysql.jdbc.JDBC4DatabaseMetaData.<init>(JDBC4DatabaseMetaData.java:39) > > at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown > Source) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.java:657) > > at > com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3064) > > at > com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3056) > > at > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2315) > > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) > > at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) > > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) > > at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) > > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) > > at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) > > at > org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208) > > at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) > > at org.hibernate.loader.Loader.doQuery(Loader.java:802) > > at > org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274) > > at org.hibernate.loader.Loader.doList(Loader.java:2542) > > at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276) > > at org.hibernate.loader.Loader.list(Loader.java:2271) > > at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:459) > > at > org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:365) > > at > org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196) > > at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268) > > at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) > > at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:246) > > > > ...... > > > > at > com.gg.healthcheck.servlet.HealthCheckServlet.getServiceStatus(HealthCheckServlet.java:68) > > at > com.gg.healthcheck.servlet.HealthCheckServlet.doGet(HealthCheckServlet.java:45) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > > at > org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) > > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) > > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) > > at > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) > > at > org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615) > > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > > at > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) > > at org.apache.tomcat.util.net > .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1626) > > at org.apache.tomcat.util.net > .SocketProcessorBase.run(SocketProcessorBase.java:49) > > - locked org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper@526aa725 > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:745) > > > > Locked synchronizers: count = 1 > > - java.util.concurrent.ThreadPoolExecutor$Worker@21c6b5b9 > > > > > > > > > > > > Tomcat version : 8.8.57 > > Java 7 > > Application is a simple spring mvc application. > > Runs on Kubernetes > > > > > > > > Do you have any idea how to solve it? > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Gokhan Akgul Best regards
