-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Greg,
On 12/22/16 12:50 PM, Greg Huber wrote: > Thanks, I have been running this setup for ages, 8 plus years and > various tomcat versions. It has never failed on <= 8.0.x Interesting. Usually, things don't just stop working like that. Did you change anything besides replacing Tomcat 8.0.x with 8.5.x? > I had to stop/start a couple of times to get it to fail and had to > run a parallel maven build to slow things down. Does that "slow down" cause your database to slow-down as well? Long-running queries can pile-up on top of one another sometimes. In dev, I always run with pool size=1 so I can catch problem early. Is it possible that you have a very small pool size and even a little bit of load can bring everything to a crawl? > Do not read much on dumps, but what would this -locked be? > > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoi nt.java:1437) > > at > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase .java:49) > > - - locked <0x00000000f1c54ed8> (a > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:617) > > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThr ead.java:61) > > at java.lang.Thread.run(Thread.java:745) "Locked" means that the thread holds a lock on a monitor. Basically, the thread has synchronized on something and it's telling you what the object id is (I think that's just a pointer, actually) and its type (an NioSocketWrapper). You'll see the same thing for the datasource: a thread has locked the data source's latch object while it waits for a connection to become available. - -chris > On 22 December 2016 at 16:24, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Greg, > > On 12/22/16 5:18 AM, Greg Huber wrote: >>>> Here is the first dump and I cannot send all three. Its my >>>> prod server which is locked up at the moment. I need to get >>>> it back on line so is this enough info? I can emile the >>>> whole file which has the three dumps if needed. >>>> >>>> [snip] >>>> >>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 >>>> mixed mode): >>>> >>>> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0 >>>> tid=0x000000000cb10000 nid=0x1960 in Object.wait() >>>> [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f2879c10> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Waiting on a database connection. > > >>>> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0 >>>> tid=0x000000000c74f800 nid=0x195d in Object.wait() >>>> [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f2852128> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Another thread waiting for a db connection. > >>>> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0 >>>> tid=0x000000000dd3b000 nid=0x1959 in Object.wait() >>>> [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f282a818> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > I'm starting to see a pattern, here. > >>>> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0 >>>> tid=0x000000000cbe6000 nid=0x1953 in Object.wait() >>>> [0x00002ba658988000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f28030f8> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Hmm. Could it be... > >>>> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0 >>>> tid=0x000000000e797800 nid=0x194f in Object.wait() >>>> [0x00002ba659472000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f27daf28> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > ... just possibly ... > >>>> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0 >>>> tid=0x000000000e874000 nid=0x194d in Object.wait() >>>> [0x00002ba656c67000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f27b3d20> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > ... that you have run out of ... > >>>> "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55 >>>> prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting >>>> on condition [0x00002ba658c50000] java.lang.Thread.State: >>>> TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native >>>> Method) at >>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(Job Sto > >>>> reSupport.java:3992) > > (Quartz >>>> > waiting around for some event. Ignore this one in erms of my > overarching narrative.) > >>>> "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0 >>>> tid=0x000000000e44f000 nid=0x194a in Object.wait() >>>> [0x00002ba658948000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f1f541b8> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Database connections? > >>>> "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0 >>>> tid=0x000000000cc2e800 nid=0x1940 in Object.wait() >>>> [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36 ) > >>>> > Ooh! >>>> > This one is doing something different. I suspect any ms it will ask > for another db connection. > >>>> "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0 >>>> tid=0x000000000cc7b800 nid=0x1933 in Object.wait() >>>> [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f1d766c0> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > That's more like it. > >>>> "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0 >>>> tid=0x000000000cb83000 nid=0x1930 in Object.wait() >>>> [0x00002ba658885000] java.lang.Thread.State: WAITING (on >>>> object monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic Obj > >>>> ectPool.java:1104) >>>> >>>> > - locked <0x00000000f1d4e918> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > I'm reminded of a Far Side cartoon. > [http://grahammercer.com.au/humour/GodMakesTheSnake.jpg] > >>>> "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0 >>>> tid=0x000000000ea8b000 nid=0x192e in Object.wait() >>>> [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36 ) > >>>> > Another >>>> > thread caught in the filter. Filters are usually quick, so I think > this is a transient state. Probably about to request a database > connection. > >>>> "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0 >>>> tid=0x000000000c92a000 nid=0x1928 in Object.wait() >>>> [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36 ) > >>>> > Here, >>>> > too. > > > Okay, I'm done reading the thread dump. > > Have a look at your database pool's size and "active" counts[1]. > You may find that your application is locked-up waiting on db > connections. you may have a resource leak (db connections). > > -chris > > [1] > http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20A pa > > che%20Tomcat%20with%20JMX.pdf, > slide 16 >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJYXB+lAAoJEBzwKT+lPKRYwx8QALOUcVOjkG9HM34b+JYjpCtP pqNolieQShhcyqCIBHmtfX/ftdtbbhfEvJq2VlOXr/Bos94sA5dVFG1K5FMepJ5b EZKON1sQp8FPRqahs9nsOoQVYivV/+6DO4gPKbVtRedy+qbOlqFvnLJGpyXTew7s 3ljMxRfgqppy12B2KvXnOCG07Zbr8l9GNWPJja7fxVTax9UA4CE2LHmsx9W7ZdqW WCmo3L+ELuF4ZLwPEaYat6sSrrrMNx47xFv5oXxjE92whyNnILGzULpnNZKIFki6 HLVj/TTbyff0pygUeztIJV9+DuWHw05LGi0WB7G6YZ6bxy7bFliPBLmG8HvjPWd0 AiC5OtQ13FYauRPeW7jn/4HeUjvrRWTmwRNDn+AbUlHvSVrDeET5pPENltC5KzF2 ja/1MAFvyQECY4jGIqHTn5IWICmjHwnP2PFPqTN6AUiDHqheKT2oemWAWPxmesdd 5zn5igU8AI6N9dZrXpniDOiiK+YaECmBdPfrPnBx/p5Ni4Ne+HTJNE4YywVtRNb2 2fz9DaOlFTlD+FkbyVqRBgJQT1m6ygfyQ0+PUf6FS80mOwmPE9C9QkKAFgOui046 NdZ7viClzIiCmsmR1JXgVkV+LAfmcZfkwjU+zBDmbLu7/G1Y8Q8U2tFRdvax1n8f vN4WY1nm3xQEqbUL5jI6 =Mp0Q -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
