Chris,
Thanks, I have been running this setup for ages, 8 plus years and various
tomcat versions. It has never failed on <= 8.0.x I had to stop/start a
couple of times to get it to fail and had to run a parallel maven build to
slow things down.
Do not read much on dumps, but what would this -locked be?
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000000f1c54ed8> (a
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Cheers Greg
On 22 December 2016 at 16:24, Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Greg,
>
> On 12/22/16 5:18 AM, Greg Huber wrote:
> > Here is the first dump and I cannot send all three. Its my prod
> > server which is locked up at the moment. I need to get it back on
> > line so is this enough info? I can emile the whole file which has
> > the three dumps if needed.
> >
> > [snip]
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> > mode):
> >
> > "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
> > tid=0x000000000cb10000 nid=0x1960 in Object.wait()
> > [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f2879c10> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> Waiting on a database connection.
>
>
> > "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
> > tid=0x000000000c74f800 nid=0x195d in Object.wait()
> > [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f2852128> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> Another thread waiting for a db connection.
>
> > "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
> > tid=0x000000000dd3b000 nid=0x1959 in Object.wait()
> > [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f282a818> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> I'm starting to see a pattern, here.
>
> > "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
> > tid=0x000000000cbe6000 nid=0x1953 in Object.wait()
> > [0x00002ba658988000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f28030f8> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> Hmm. Could it be...
>
> > "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
> > tid=0x000000000e797800 nid=0x194f in Object.wait()
> > [0x00002ba659472000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f27daf28> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> ... just possibly ...
>
> > "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0
> > tid=0x000000000e874000 nid=0x194d in Object.wait()
> > [0x00002ba656c67000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f27b3d20> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> ... that you have run out of ...
>
> > "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55
> > prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting on
> > condition [0x00002ba658c50000] java.lang.Thread.State:
> > TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method)
> > at
> > org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobSto
> reSupport.java:3992)
>
> (Quartz
> >
> waiting around for some event. Ignore this one in erms of my
> overarching narrative.)
>
> > "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0
> > tid=0x000000000e44f000 nid=0x194a in Object.wait()
> > [0x00002ba658948000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f1f541b8> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> Database connections?
>
> > "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0
> > tid=0x000000000cc2e800 nid=0x1940 in Object.wait()
> > [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at
> > org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)
>
> Ooh!
> >
> This one is doing something different. I suspect any ms it will
> ask for another db connection.
>
> > "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0
> > tid=0x000000000cc7b800 nid=0x1933 in Object.wait()
> > [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f1d766c0> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> That's more like it.
>
> > "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0
> > tid=0x000000000cb83000 nid=0x1930 in Object.wait()
> > [0x00002ba658885000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0x00000000f1d4e918> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> I'm reminded of a Far Side cartoon.
> [http://grahammercer.com.au/humour/GodMakesTheSnake.jpg]
>
> > "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0
> > tid=0x000000000ea8b000 nid=0x192e in Object.wait()
> > [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at
> > org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)
>
> Another
> >
> thread caught in the filter. Filters are usually quick, so I
> think this is a transient state. Probably about to request a database
> connection.
>
> > "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0
> > tid=0x000000000c92a000 nid=0x1928 in Object.wait()
> > [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at
> > org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)
>
> Here,
> >
> too.
>
>
> Okay, I'm done reading the thread dump.
>
> Have a look at your database pool's size and "active" counts[1]. You
> may find that your application is locked-up waiting on db connections.
> you may have a resource leak (db connections).
>
> - -chris
>
> [1]
> http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20Apa
> che%20Tomcat%20with%20JMX.pdf,
> slide 16
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYW/4vAAoJEBzwKT+lPKRYj8QP/jB4YkmoyLwS6z1RZ4XnV9Gw
> 43b2aoSdhDihonAao+Zu23+PQCELkzhv06hA2xZ33fW+HFem7cMw3isVh1NTxtVO
> 1fKsS4ZVftMnjHrFSMN9dzWnClf2vnDgN78to/tsEJGkinP7wJL55qZkdETJF4FO
> L3vMabMT1LeOUo8bdJfYKJXnA2vrjoU7W8vLP883LZy3YMUCdVclFOG10v/kr7vB
> oSWtSXBlq83fFJasXCSXEH7YvLteY1+jqrOx87JTeKgnYIkBF1KujMKuqw0ADRig
> brYegISWAdjs9ovyeTh4PKyWc29lNp7jwRcEMOwh1XF7WgV1msW3awqdIdPnNrtt
> 1kKLNXJv/aKILwMRCWQyR9/2jCsQygOE3Q5qXAjkjMe8qfORZYDdPQlZzpEuGt3U
> IMrRbpUN+jhMLKtfs/PWsuCker5rdGIIQu/SkCiYqpSDPiRpXLTteOW82SjxvLN/
> QrasunCCz3N3Y0XEFJbkuE49lNYRruYuhEpmxXYuB+Xv6GOQzic9FYfztwq9z41S
> AvOjCiTCSacpScPoB9k3q8YGdxZniKQG6pgm0p+NoQYTO8mid/gznftr35GO+Dw8
> c1zREZd5qv858CON1ELthxTFSn43Bte7bxfFl9ZvZ8DOWXEYi8P7fDJcuoBDoxuK
> Ag1tPGX3fMKaeC5Bwyw9
> =ogwm
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
