On 24/12/2016 08:49, Greg Huber wrote: > ...Well its not hibernate, will look at switching to commons-dbcp2 if > possible.
Again, please post your resource config. Mark > > Cheers > > On 22 December 2016 at 20:26, Mark Thomas <ma...@apache.org> wrote: > >> On 22/12/2016 17:50, Greg Huber wrote: >>> Chris, >>> >>> Thanks, I have been running this setup for ages, 8 plus years and various >>> tomcat versions. It has never failed on <= 8.0.x I had to stop/start a >>> couple of times to get it to fail and had to run a parallel maven build >> to >>> slow things down. >>> >>> Do not read much on dumps, but what would this -locked be? >> >> It is perfectly normal. >> >> Your problem is with the connection pool. >> >> Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of >> the attributes changed. You might not be using the settings you think >> you are using. Please post your resource config. >> >> Mark >> >>> >>> at >>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor. >> doRun(NioEndpoint.java:1437) >>> at >>> org.apache.tomcat.util.net.SocketProcessorBase.run( >> SocketProcessorBase.java:49) >>> - locked <0x00000000f1c54ed8> (a >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >>> at >>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run( >> TaskThread.java:61) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Cheers Greg >>> >>> >>> On 22 December 2016 at 16:24, Christopher Schultz < >>> ch...@christopherschultz.net> wrote: >>> >>> Greg, >>> >>> On 12/22/16 5:18 AM, Greg Huber wrote: >>>>>> Here is the first dump and I cannot send all three. Its my prod >>>>>> server which is locked up at the moment. I need to get it back on >>>>>> line so is this enough info? I can emile the whole file which has >>>>>> the three dumps if needed. >>>>>> >>>>>> [snip] >>>>>> >>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed >>>>>> mode): >>>>>> >>>>>> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000cb10000 nid=0x1960 in Object.wait() >>>>>> [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f2879c10> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> Waiting on a database connection. >>> >>> >>>>>> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000c74f800 nid=0x195d in Object.wait() >>>>>> [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f2852128> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> Another thread waiting for a db connection. >>> >>>>>> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000dd3b000 nid=0x1959 in Object.wait() >>>>>> [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f282a818> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> I'm starting to see a pattern, here. >>> >>>>>> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000cbe6000 nid=0x1953 in Object.wait() >>>>>> [0x00002ba658988000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f28030f8> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> Hmm. Could it be... >>> >>>>>> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000e797800 nid=0x194f in Object.wait() >>>>>> [0x00002ba659472000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f27daf28> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> ... just possibly ... >>> >>>>>> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000e874000 nid=0x194d in Object.wait() >>>>>> [0x00002ba656c67000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f27b3d20> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> ... that you have run out of ... >>> >>>>>> "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55 >>>>>> prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting on >>>>>> condition [0x00002ba658c50000] java.lang.Thread.State: >>>>>> TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) >>>>>> at >>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$ >> MisfireHandler.run(JobSto >>> reSupport.java:3992) >>> >>> (Quartz >>>>>> >>> waiting around for some event. Ignore this one in erms of my >>> overarching narrative.) >>> >>>>>> "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000e44f000 nid=0x194a in Object.wait() >>>>>> [0x00002ba658948000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f1f541b8> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> Database connections? >>> >>>>>> "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000cc2e800 nid=0x1940 in Object.wait() >>>>>> [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at >>>>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) >>> >>> Ooh! >>>>>> >>> This one is doing something different. I suspect any ms it will >>> ask for another db connection. >>> >>>>>> "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000cc7b800 nid=0x1933 in Object.wait() >>>>>> [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f1d766c0> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> That's more like it. >>> >>>>>> "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000cb83000 nid=0x1930 in Object.wait() >>>>>> [0x00002ba658885000] java.lang.Thread.State: WAITING (on object >>>>>> monitor) at java.lang.Object.wait(Native Method) at >>>>>> java.lang.Object.wait(Object.java:502) at >>>>>> org.apache.commons.pool.impl.GenericObjectPool. >> borrowObject(GenericObj >>> ectPool.java:1104) >>>>>> >>>>>> >>> - locked <0x00000000f1d4e918> (a >>>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) >>> >>> I'm reminded of a Far Side cartoon. >>> [http://grahammercer.com.au/humour/GodMakesTheSnake.jpg] >>> >>>>>> "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000ea8b000 nid=0x192e in Object.wait() >>>>>> [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at >>>>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) >>> >>> Another >>>>>> >>> thread caught in the filter. Filters are usually quick, so I >>> think this is a transient state. Probably about to request a database >>> connection. >>> >>>>>> "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0 >>>>>> tid=0x000000000c92a000 nid=0x1928 in Object.wait() >>>>>> [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at >>>>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) >>> >>> Here, >>>>>> >>> too. >>> >>> >>> Okay, I'm done reading the thread dump. >>> >>> Have a look at your database pool's size and "active" counts[1]. You >>> may find that your application is locked-up waiting on db connections. >>> you may have a resource leak (db connections). >>> >>> -chris >>> >>> [1] >>> http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20Apa >>> che%20Tomcat%20with%20JMX.pdf, >>> slide 16 >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
