...Well its not hibernate, will look at switching to commons-dbcp2 if possible.
Cheers On 22 December 2016 at 20:26, Mark Thomas <ma...@apache.org> wrote: > On 22/12/2016 17:50, Greg Huber wrote: > > Chris, > > > > Thanks, I have been running this setup for ages, 8 plus years and various > > tomcat versions. It has never failed on <= 8.0.x I had to stop/start a > > couple of times to get it to fail and had to run a parallel maven build > to > > slow things down. > > > > Do not read much on dumps, but what would this -locked be? > > It is perfectly normal. > > Your problem is with the connection pool. > > Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of > the attributes changed. You might not be using the settings you think > you are using. Please post your resource config. > > Mark > > > > > at > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor. > doRun(NioEndpoint.java:1437) > > at > > org.apache.tomcat.util.net.SocketProcessorBase.run( > SocketProcessorBase.java:49) > > - locked <0x00000000f1c54ed8> (a > > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run( > TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:745) > > > > Cheers Greg > > > > > > On 22 December 2016 at 16:24, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Greg, > > > > On 12/22/16 5:18 AM, Greg Huber wrote: > >>>> Here is the first dump and I cannot send all three. Its my prod > >>>> server which is locked up at the moment. I need to get it back on > >>>> line so is this enough info? I can emile the whole file which has > >>>> the three dumps if needed. > >>>> > >>>> [snip] > >>>> > >>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed > >>>> mode): > >>>> > >>>> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0 > >>>> tid=0x000000000cb10000 nid=0x1960 in Object.wait() > >>>> [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f2879c10> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > Waiting on a database connection. > > > > > >>>> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0 > >>>> tid=0x000000000c74f800 nid=0x195d in Object.wait() > >>>> [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f2852128> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > Another thread waiting for a db connection. > > > >>>> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0 > >>>> tid=0x000000000dd3b000 nid=0x1959 in Object.wait() > >>>> [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f282a818> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > I'm starting to see a pattern, here. > > > >>>> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0 > >>>> tid=0x000000000cbe6000 nid=0x1953 in Object.wait() > >>>> [0x00002ba658988000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f28030f8> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > Hmm. Could it be... > > > >>>> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0 > >>>> tid=0x000000000e797800 nid=0x194f in Object.wait() > >>>> [0x00002ba659472000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f27daf28> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > ... just possibly ... > > > >>>> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0 > >>>> tid=0x000000000e874000 nid=0x194d in Object.wait() > >>>> [0x00002ba656c67000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f27b3d20> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > ... that you have run out of ... > > > >>>> "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55 > >>>> prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting on > >>>> condition [0x00002ba658c50000] java.lang.Thread.State: > >>>> TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) > >>>> at > >>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$ > MisfireHandler.run(JobSto > > reSupport.java:3992) > > > > (Quartz > >>>> > > waiting around for some event. Ignore this one in erms of my > > overarching narrative.) > > > >>>> "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0 > >>>> tid=0x000000000e44f000 nid=0x194a in Object.wait() > >>>> [0x00002ba658948000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f1f541b8> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > Database connections? > > > >>>> "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0 > >>>> tid=0x000000000cc2e800 nid=0x1940 in Object.wait() > >>>> [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at > >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) > > > > Ooh! > >>>> > > This one is doing something different. I suspect any ms it will > > ask for another db connection. > > > >>>> "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0 > >>>> tid=0x000000000cc7b800 nid=0x1933 in Object.wait() > >>>> [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f1d766c0> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > That's more like it. > > > >>>> "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0 > >>>> tid=0x000000000cb83000 nid=0x1930 in Object.wait() > >>>> [0x00002ba658885000] java.lang.Thread.State: WAITING (on object > >>>> monitor) at java.lang.Object.wait(Native Method) at > >>>> java.lang.Object.wait(Object.java:502) at > >>>> org.apache.commons.pool.impl.GenericObjectPool. > borrowObject(GenericObj > > ectPool.java:1104) > >>>> > >>>> > > - locked <0x00000000f1d4e918> (a > >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > > > I'm reminded of a Far Side cartoon. > > [http://grahammercer.com.au/humour/GodMakesTheSnake.jpg] > > > >>>> "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0 > >>>> tid=0x000000000ea8b000 nid=0x192e in Object.wait() > >>>> [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at > >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) > > > > Another > >>>> > > thread caught in the filter. Filters are usually quick, so I > > think this is a transient state. Probably about to request a database > > connection. > > > >>>> "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0 > >>>> tid=0x000000000c92a000 nid=0x1928 in Object.wait() > >>>> [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at > >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) > > > > Here, > >>>> > > too. > > > > > > Okay, I'm done reading the thread dump. > > > > Have a look at your database pool's size and "active" counts[1]. You > > may find that your application is locked-up waiting on db connections. > > you may have a resource leak (db connections). > > > > -chris > > > > [1] > > http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20Apa > > che%20Tomcat%20with%20JMX.pdf, > > slide 16 > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
