Andrea Peruffo created ZEPPELIN-2502:
----------------------------------------
Summary: RemoteInterpreterServer hang forever during shutdown
Key: ZEPPELIN-2502
URL: https://issues.apache.org/jira/browse/ZEPPELIN-2502
Project: Zeppelin
Issue Type: Bug
Components: zeppelin-interpreter
Affects Versions: 0.7.1, 0.7.0, 0.7.2, 0.8.0
Reporter: Andrea Peruffo
TL;DR;
there is the opportunity that RemoteServerInterpreter does never terminate
during shutdown causing zombies JVMs around.
a proper Unit test is provided here:
https://github.com/nokia/zeppelin/blob/35161ba8595782275a1f597f9267a92c226a6cc9/zeppelin-interpreter/src/test/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServerTest.java#L92
VERSION:
bug detected on 0.7.0 but applicable on master
DETAILED:
I was digging a problem with zombie JVM process caused by the interaction of
long running Spark jobs in the cluster and Spark interpreter restarts done via
UI.
I managed to reproduce a scenario where a SparkSubmit process never ends, and
analyzing the stack trace I ended up finding potential conditions where I can
have zombies in RemoteInterpreter.
I'm still not 100% sure this will solve but it is anyhow a fix to a bug that
can manifest itself sooner or later.
My stack trace of the hanging process was:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode):
"Attach Listener" #118 daemon prio=9 os_prio=0 tid=0x00007fb03c001000
nid=0x104e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"pool-2-thread-6" #114 prio=5 os_prio=0 tid=0x00007fb00c004000 nid=0xe52
waiting on condition [0x00007faf963c1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00da470> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"pool-2-thread-5" #108 prio=5 os_prio=0 tid=0x00007fb01403d000 nid=0xd8d
waiting on condition [0x00007faf966c4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00da470> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"pool-2-thread-4" #107 prio=5 os_prio=0 tid=0x00007fb01403c000 nid=0xd8c in
Object.wait() [0x00007faf967c5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:112)
- locked <0x00000000dc206700> (a java.util.LinkedList)
- locked <0x00000000dc206700> (a java.util.LinkedList)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000db13c610> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-2-thread-3" #105 prio=5 os_prio=0 tid=0x00007fb00c003000 nid=0xd8a in
Object.wait() [0x00007fafa7cfd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:112)
- locked <0x00000000dc206838> (a java.util.LinkedList)
- locked <0x00000000dc206838> (a java.util.LinkedList)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000db13c7f8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #90
daemon prio=5 os_prio=0 tid=0x00007fb00af63000 nid=0xccd in Object.wait()
[0x00007faf9cfe1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x00000000cce05198> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"BoneCP-pool-watch-thread" #89 daemon prio=5 os_prio=0 tid=0x00007fb009fdd000
nid=0xccc waiting on condition [0x00007faf9d8e2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000cd439380> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000cd439468> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"BoneCP-keep-alive-scheduler" #88 daemon prio=5 os_prio=0
tid=0x00007fb009fdb800 nid=0xccb waiting on condition [0x00007faf9d9e3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000cd43fdf0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"com.google.common.base.internal.Finalizer" #87 daemon prio=5 os_prio=0
tid=0x00007fb009fda000 nid=0xcca in Object.wait() [0x00007faf9dae4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x00000000cd4881a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:132)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"BoneCP-pool-watch-thread" #86 daemon prio=5 os_prio=0 tid=0x00007fb00ad1f000
nid=0xcc9 waiting on condition [0x00007faf9dfe5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000cd4891e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000cd4892c8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"BoneCP-keep-alive-scheduler" #85 daemon prio=5 os_prio=0
tid=0x00007fb00ad21000 nid=0xcc8 waiting on condition [0x00007faf9e0e6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000cd489c18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"com.google.common.base.internal.Finalizer" #84 daemon prio=5 os_prio=0
tid=0x00007fb00ad1d000 nid=0xcc7 in Object.wait() [0x00007faf9e1e7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x00000000cd48a528> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:132)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"derby.rawStoreDaemon" #83 daemon prio=5 os_prio=0 tid=0x00007fb00acb8800
nid=0xcc6 in Object.wait() [0x00007faf9e4e8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown
Source)
- locked <0x00000000cceb07f8> (a
org.apache.derby.impl.services.daemon.BasicDaemon)
at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Timer-1" #82 daemon prio=5 os_prio=0 tid=0x00007fb00ac4f800 nid=0xcc5 in
Object.wait() [0x00007faf9e7e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000cd3b9f08> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"pool-1-thread-5" #81 prio=5 os_prio=0 tid=0x00007fb024008800 nid=0xcc1
runnable [0x00007faf9f8ea000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000cd48ac30> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000cd48ad40> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-2-thread-2" #15 prio=5 os_prio=0 tid=0x00007fb00c002000 nid=0xc72 waiting
on condition [0x00007fb013ffe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00da470> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007fb0200b4000 nid=0xc71 in
Object.wait() [0x00007fb0281ca000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:112)
- locked <0x00000000c0003a20> (a java.util.LinkedList)
- locked <0x00000000c0003a20> (a java.util.LinkedList)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000c0003b18> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-1-thread-4" #13 prio=5 os_prio=0 tid=0x00007fb024006800 nid=0xc70 waiting
on condition [0x00007fb0286dc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0004058> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"pool-1-thread-3" #12 prio=5 os_prio=0 tid=0x00007fb024005800 nid=0xc6f
runnable [0x00007fb0287dd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000c00ff608> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000c0007180> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-1-thread-2" #11 prio=5 os_prio=0 tid=0x00007fb02400a800 nid=0xc6e in
Object.wait() [0x00007fb0288de000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterpreterEventClient.java:220)
- locked <0x00000000c0003cc0> (a java.util.LinkedList)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterServer.java:791)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1863)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1848)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000c000d070> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-1-thread-1" #10 prio=5 os_prio=0 tid=0x00007fb024003800 nid=0xc6d
runnable [0x00007fb0289df000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000d95c4508> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000c0004210> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"Thread-0" #9 prio=5 os_prio=0 tid=0x00007fb07cf75800 nid=0xc6a runnable
[0x00007fb028ae0000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:96)
Locked ownable synchronizers:
- None
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fb07c0d1000 nid=0xc68
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fb07c0bb800
nid=0xc67 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fb07c0b9800
nid=0xc66 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fb07c0b6800
nid=0xc65 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fb07c0b4800 nid=0xc64
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fb07c086000 nid=0xc63 in
Object.wait() [0x00007fb0513f2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x00000000c00028f0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fb07c084000
nid=0xc62 in Object.wait() [0x00007fb0514f3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000000c00030f0> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00007fb07c00f000 nid=0xc5a in Object.wait()
[0x00007fb083f1c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c0003d00> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
at java.lang.Thread.join(Thread.java:1245)
- locked <0x00000000c0003d00> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
at java.lang.Thread.join(Thread.java:1319)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007fb07c07f000 nid=0xc61 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fb07c024800 nid=0xc5b
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fb07c026000 nid=0xc5c
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fb07c028000 nid=0xc5d
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fb07c02a000 nid=0xc5e
runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007fb07c02b800 nid=0xc5f
runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007fb07c02d800 nid=0xc60
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fb07c0db800 nid=0xc69 waiting on
condition
JNI global references: 288
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
