>1) How do I enable DEBUG logging on the Ignite cluster? It depends on the logger that you use. You have to change the config file and give to the ignite package DEBUG level. It could be JUL, log4j or log4j2 config files.
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology Client node didn't even join the topology. Most possible you have incorrect discovery configuration. Also, I'd recommend to check client logs for more information why node can't connect to the cluster. Evgenii вс, 4 нояб. 2018 г. в 8:09, sv <[email protected]>: > Evgenii, > > 1) How do I enable DEBUG logging on the Ignite cluster? > > 2) I used jstack to produce a thread dump of the Zeppelin Ignite > interpreter > (which runs in a separate process). I pasted it below. Does anything look > out of place to you? > > 2018-11-03 09:32:37 > Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode): > > "Attach Listener" #82 daemon prio=9 os_prio=0 tid=0x00007f202c001000 > nid=0x37b waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "tcp-client-disco-msg-worker-#4" #81 prio=5 os_prio=0 > tid=0x00007f1fd1f2a000 > nid=0x2fb waiting on condition [0x00007f1fce1e6000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > > org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:522) > at > > org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128) > at > > org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846) > at > > org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1561) > at > org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > Locked ownable synchronizers: > - None > > "tcp-client-disco-sock-reader-#3" #80 prio=5 os_prio=0 > tid=0x00007f1fd1f25000 nid=0x2fa in Object.wait() [0x00007f1fce2e7000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at > > org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:1006) > - locked <0x00000000ee2bc268> (a java.lang.Object) > at > org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > Locked ownable synchronizers: > - None > > "tcp-client-disco-sock-writer-#2" #79 prio=5 os_prio=0 > tid=0x00007f1fd1f24800 nid=0x2f9 in Object.wait() [0x00007f1fce3e8000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at > > org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1209) > - locked <0x00000000ee2bbd88> (a java.lang.Object) > at > org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > Locked ownable synchronizers: > - None > > "TcpDiscoverySpi.timer" #78 prio=5 os_prio=0 tid=0x00007f1fd1f20800 > nid=0x2f8 in Object.wait() [0x00007f1fce4e9000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.util.TimerThread.mainLoop(Timer.java:526) > - locked <0x00000000ee228468> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:505) > > Locked ownable synchronizers: > - None > > "grid-data-loader-flusher-#39" #77 prio=5 os_prio=0 tid=0x00007f1fd1efb000 > nid=0x2f7 waiting on condition [0x00007f1fce5ea000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000edc52cc0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) > at > > org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-client-listener-3-#37" #75 prio=5 os_prio=0 > tid=0x00007f1fd1eea800 nid=0x2f6 runnable [0x00007f1fce8eb000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000edb0fdf0> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000ed992ff0> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000ed992ed8> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-client-listener-2-#36" #74 prio=5 os_prio=0 > tid=0x00007f1fd1ee8800 nid=0x2f5 runnable [0x00007f1fce9ec000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000ed98e750> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000ed97e4f8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000ed97e3e0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-client-listener-1-#35" #73 prio=5 os_prio=0 > tid=0x00007f1fd1ee7000 nid=0x2f4 runnable [0x00007f1fceaed000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000ed979c58> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000ed969a00> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000ed9698e8> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-client-listener-0-#34" #72 prio=5 os_prio=0 > tid=0x00007f1fd1e67800 nid=0x2f3 runnable [0x00007f1fcebee000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000ed965160> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000ed954f08> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000ed954df0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "nio-acceptor-#33" #71 prio=5 os_prio=0 tid=0x00007f1fd1e66800 nid=0x2f2 > runnable [0x00007f1fcecef000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000ed944218> (a sun.nio.ch.Util$3) > - locked <0x00000000ed944208> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000ed9440f0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870) > at > > org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "tcp-comm-worker-#1" #70 prio=5 os_prio=0 tid=0x00007f1fd1cdf800 nid=0x2f1 > waiting on condition [0x00007f1fcf1f0000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000fd641060> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4207) > at > org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-comm-3-#32" #66 prio=5 os_prio=0 > tid=0x00007f1fd1cdd800 > nid=0x2f0 runnable [0x00007f1fcf2f1000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000fda143b0> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000fda16430> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000fda14308> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-comm-2-#31" #65 prio=5 os_prio=0 > tid=0x00007f1fd1cd7000 > nid=0x2ef runnable [0x00007f1fcf6f5000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000fda16b38> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000fda49580> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000fda16a90> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-comm-1-#30" #64 prio=5 os_prio=0 > tid=0x00007f1fd1cd8800 > nid=0x2ee runnable [0x00007f1fcf4f3000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000fda49c88> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000fda4bd08> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000fda49be0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-comm-0-#29" #63 prio=5 os_prio=0 > tid=0x00007f1fd1cd5800 > nid=0x2ed runnable [0x00007f1fcf5f4000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000fda4c410> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000fda4e490> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000fda4c368> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "nio-acceptor-#28" #62 prio=5 os_prio=0 tid=0x00007f1fd1cdb000 nid=0x2ec > runnable [0x00007f1fcf3f2000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000fda13c60> (a sun.nio.ch.Util$3) > - locked <0x00000000fda13c50> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000fda13838> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870) > at > > org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-timeout-worker-#27" #61 prio=5 os_prio=0 tid=0x00007f1fd1c4b800 > nid=0x2e8 in Object.wait() [0x00007f1fcf7f6000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at > > org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:203) > - locked <0x00000000fd9add58> (a java.lang.Object) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "ignite-clock" #60 daemon prio=10 os_prio=0 tid=0x00007f1fd1c3f000 > nid=0x2e7 > sleeping[0x00007f1fcf8f7000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3308) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "jvm-pause-detector-worker" #53 daemon prio=5 os_prio=0 > tid=0x00007f1fd184e800 nid=0x2e0 sleeping[0x00007f1fcfffe000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > > org.apache.ignite.internal.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-7-#20" #52 prio=5 os_prio=0 tid=0x00007f1fd1805800 > nid=0x2df waiting on condition [0x00007f1ff836b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-6-#19" #51 prio=5 os_prio=0 tid=0x00007f1fd1803800 > nid=0x2de waiting on condition [0x00007f1ff846c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-5-#18" #50 prio=5 os_prio=0 tid=0x00007f1fd1802000 > nid=0x2dd waiting on condition [0x00007f1ff856d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-4-#17" #49 prio=5 os_prio=0 tid=0x00007f1fd1800000 > nid=0x2dc waiting on condition [0x00007f1ff866e000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-3-#16" #48 prio=5 os_prio=0 tid=0x00007f1fd17fe000 > nid=0x2db waiting on condition [0x00007f1ff876f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-2-#15" #47 prio=5 os_prio=0 tid=0x00007f1fd17fc000 > nid=0x2da waiting on condition [0x00007f1ff8870000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-1-#14" #46 prio=5 os_prio=0 tid=0x00007f1fd17fa800 > nid=0x2d9 waiting on condition [0x00007f1ff8971000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "data-streamer-stripe-0-#13" #45 prio=5 os_prio=0 tid=0x00007f1fd17f8800 > nid=0x2d8 waiting on condition [0x00007f1ff8a72000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-7-#12" #44 prio=5 os_prio=0 tid=0x00007f1fd17f6800 nid=0x2d7 > waiting on condition [0x00007f1ff8b73000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-6-#11" #43 prio=5 os_prio=0 tid=0x00007f1fd17f4800 nid=0x2d6 > waiting on condition [0x00007f1ff8c74000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-5-#10" #42 prio=5 os_prio=0 tid=0x00007f1fd17f2800 nid=0x2d5 > waiting on condition [0x00007f1ff8d75000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-4-#9" #41 prio=5 os_prio=0 tid=0x00007f1fd17f0800 nid=0x2d4 > waiting on condition [0x00007f1ff8e76000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-3-#8" #40 prio=5 os_prio=0 tid=0x00007f1fd17ee800 nid=0x2d3 > waiting on condition [0x00007f1ff8f77000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-2-#7" #39 prio=5 os_prio=0 tid=0x00007f1fd17d3000 nid=0x2d2 > waiting on condition [0x00007f1ff9078000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-1-#6" #38 prio=5 os_prio=0 tid=0x00007f1fd17d1800 nid=0x2d1 > waiting on condition [0x00007f1ff9179000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "sys-stripe-0-#5" #37 prio=5 os_prio=0 tid=0x00007f1fd17cf000 nid=0x2d0 > waiting on condition [0x00007f1ffa0f1000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > > org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672) > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "JobProgressPoller, jobId=20181102-071721_1550658378" #36 prio=5 os_prio=0 > tid=0x00007f1fd0003000 nid=0x2cf waiting on condition [0x00007f1ffadf2000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > > org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:61) > > Locked ownable synchronizers: > - None > > "pool-2-thread-7" #35 prio=5 os_prio=0 tid=0x00007f2008001000 nid=0x2ce > waiting on condition [0x00007f1ffaef3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0006268> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "pool-2-thread-6" #33 prio=5 os_prio=0 tid=0x00007f1ffc005800 nid=0x2c9 > waiting on condition [0x00007f1ffb1f4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0006268> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "pool-2-thread-5" #31 prio=5 os_prio=0 tid=0x00007f1ffc004800 nid=0x2c4 > waiting on condition [0x00007f1ffb2f4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000ee2280f0> (a > java.util.concurrent.CountDownLatch$Sync) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > at > > org.apache.ignite.spi.discovery.tcp.ClientImpl.spiStart(ClientImpl.java:271) > at > > org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948) > at > > org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) > at > > org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915) > at > > org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721) > at > org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028) > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014) > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723) > - locked <0x00000000fd9eaa10> (a > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance) > at > org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:671) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596) > at org.apache.ignite.Ignition.start(Ignition.java:327) > at > > org.apache.zeppelin.ignite.IgniteInterpreter.getIgnite(IgniteInterpreter.java:190) > at > > org.apache.zeppelin.ignite.IgniteInterpreter.initIgnite(IgniteInterpreter.java:208) > at > > org.apache.zeppelin.ignite.IgniteInterpreter.open(IgniteInterpreter.java:127) > at > > org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) > - locked <0x00000000c000f738> (a > org.apache.zeppelin.ignite.IgniteInterpreter) > - locked <0x00000000c000f810> (a > org.apache.zeppelin.interpreter.LazyOpenInterpreter) > at > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:617) > at org.apache.zeppelin.scheduler.Job.run(Job.java:188) > at > org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - <0x00000000fda11fc8> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > - <0x00000000fda12080> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > - <0x00000000fda12138> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > - <0x00000000fda121f0> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > - <0x00000000c000f900> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > "pool-2-thread-4" #29 prio=5 os_prio=0 tid=0x00007f1ffc003000 nid=0x2bf > waiting on condition [0x00007f1ffb3f6000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0006268> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "ignite-client-exec-3-5" #28 daemon prio=5 os_prio=0 tid=0x00007f1fd0001000 > nid=0x2bb waiting on condition [0x00007f1ffb4f7000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c001bdb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "ignite-client-exec-3-4" #27 daemon prio=5 os_prio=0 tid=0x00007f1fe4001000 > nid=0x2ba waiting on condition [0x00007f2018230000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c001bdb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "ignite-client-exec-3-3" #26 daemon prio=5 os_prio=0 tid=0x00007f20001c9000 > nid=0x2b9 waiting on condition [0x00007f1ffb5f8000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c001bdb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "a65ff0b2-c2ba-4d1c-8d67-25570203a476-topology-update" #25 daemon prio=5 > os_prio=0 tid=0x00007f20001c5800 nid=0x2b8 waiting on condition > [0x00007f1ffb6f9000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > > org.apache.ignite.internal.client.impl.GridClientImpl$TopologyUpdaterThread.run(GridClientImpl.java:521) > > Locked ownable synchronizers: > - None > > "ignite-client-top-lsnr-1-2" #24 daemon prio=5 os_prio=0 > tid=0x00007f1ff000d000 nid=0x2b7 waiting on condition [0x00007f1ffb7fa000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c001c160> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "ignite-client-exec-3-2" #23 daemon prio=5 os_prio=0 tid=0x00007f20001ae000 > nid=0x2b6 waiting on condition [0x00007f1ffb8fb000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c001bdb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-client-3-#4%gridClient%" #22 prio=5 os_prio=0 > tid=0x00007f200017c800 nid=0x2b5 runnable [0x00007f1ffbbfc000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000c0007f00> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000c0007ef0> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000c0007f20> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-client-2-#3%gridClient%" #21 prio=5 os_prio=0 > tid=0x00007f200017a800 nid=0x2b4 runnable [0x00007f1ffbcfd000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000c001c610> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000c001c600> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000c001c630> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-client-1-#2%gridClient%" #20 prio=5 os_prio=0 > tid=0x00007f2000179000 nid=0x2b3 runnable [0x00007f1ffbdfe000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000c00237f8> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000c003a000> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000c00236e0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "grid-nio-worker-tcp-client-0-#1%gridClient%" #19 prio=5 os_prio=0 > tid=0x00007f2000177000 nid=0x2b2 runnable [0x00007f201812a000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000c003aa90> (a > org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) > - locked <0x00000000c003d658> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000c003a9e8> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105) > at > > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "pool-2-thread-3" #16 prio=5 os_prio=0 tid=0x00007f1ffc001000 nid=0x2b0 > waiting on condition [0x00007f2018331000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0006268> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > > "pool-2-thread-2" #15 prio=5 os_prio=0 tid=0x00007f2004007000 nid=0x2af in > Object.wait() [0x00007f2018432000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at > org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113) > - locked <0x00000000c000fb00> (a java.util.LinkedList) > - locked <0x00000000c000fb00> (a java.util.LinkedList) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - <0x00000000c00c82c8> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > "pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007f2004005000 nid=0x2ae in > Object.wait() [0x00007f2018533000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at > org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113) > - locked <0x00000000c000f958> (a java.util.LinkedList) > - locked <0x00000000c000f958> (a java.util.LinkedList) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - <0x00000000c00c85d8> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > "pool-1-thread-3" #13 prio=5 os_prio=0 tid=0x00007f2014005000 nid=0x2ad > runnable [0x00007f2018634000] > java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > - locked <0x00000000c00c8cf8> (a java.io.BufferedInputStream) > at > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > at > > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) > at > > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) > at > > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) > at > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - <0x00000000c00c8f30> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > "pool-1-thread-2" #12 prio=5 os_prio=0 tid=0x00007f2014003800 nid=0x2ac in > Object.wait() [0x00007f2018948000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterpreterEventClient.java:357) > - locked <0x00000000c00804d8> (a java.util.LinkedList) > at > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterServer.java:970) > at > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:2000) > at > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1985) > at > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - <0x00000000c00c9808> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > "pool-1-thread-1" #11 prio=5 os_prio=0 tid=0x00007f2014002000 nid=0x2ab in > Object.wait() [0x00007f2018a49000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.interpret(RemoteInterpreterServer.java:447) > - locked <0x00000000c00060b0> (a > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJobListener) > at > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1859) > at > > org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1844) > at > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - <0x00000000c00c9080> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > "Thread-0" #9 prio=5 os_prio=0 tid=0x00007f20645ef000 nid=0x2a6 runnable > [0x00007f2018b4a000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > at java.net.ServerSocket.accept(ServerSocket.java:513) > at > > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129) > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) > at > > org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60) > at > > org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161) > at > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:203) > > Locked ownable synchronizers: > - None > > "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f20640c2000 > nid=0x2a4 > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f20640bf000 > nid=0x2a3 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f20640bd000 > nid=0x2a2 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f20640ba000 > nid=0x2a1 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f20640b8800 > nid=0x2a0 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f2064086800 nid=0x29f in > Object.wait() [0x00007f2040ad0000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0x00000000c00ceaf8> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212) > > Locked ownable synchronizers: > - None > > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f2064082000 > nid=0x29e in Object.wait() [0x00007f2040bd1000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > - locked <0x00000000c00ced28> (a java.lang.ref.Reference$Lock) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > > Locked ownable synchronizers: > - None > > "main" #1 prio=5 os_prio=0 tid=0x00007f206400c800 nid=0x298 in > Object.wait() > [0x00007f206bab0000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000000c007e0f0> (a > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer) > at java.lang.Thread.join(Thread.java:1252) > - locked <0x00000000c007e0f0> (a > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer) > at java.lang.Thread.join(Thread.java:1326) > at > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:274) > > Locked ownable synchronizers: > - None > > "VM Thread" os_prio=0 tid=0x00007f206407a000 nid=0x29d runnable > > "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f2064022000 nid=0x299 > runnable > > "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f2064023800 nid=0x29a > runnable > > "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f2064025800 nid=0x29b > runnable > > "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f2064027000 nid=0x29c > runnable > > "VM Periodic Task Thread" os_prio=0 tid=0x00007f20640c4800 nid=0x2a5 > waiting > on condition > > JNI global references: 206 > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
