zhaoming.chen created ZEPPELIN-5711:
---------------------------------------
Summary: Zeppelin0.10: deadlock on zeppelin-server
Key: ZEPPELIN-5711
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5711
Project: Zeppelin
Issue Type: Bug
Components: zeppelin-server
Affects Versions: 0.10.1
Reporter: zhaoming.chen
there is deadlock in zeppelin server, which is a fatal flaw.
the jstack of zeppelin-server process is :
Found one Java-level deadlock:
=============================
"qtp1593180232-305212":
waiting to lock monitor 0x00007f541003fce8 (object 0x0000000681692280, a
java.util.HashMap),
which is held by "qtp1593180232-302665"
"qtp1593180232-302665":
waiting to lock monitor 0x00007f54100b23d8 (object 0x0000000755589698, a
org.apache.zeppelin.socket.NotebookSocket),
which is held by "qtp1593180232-302348"
"qtp1593180232-302348":
waiting to lock monitor 0x00007f541003fce8 (object 0x0000000681692280, a
java.util.HashMap),
which is held by "qtp1593180232-302665"
Java stack information for the threads listed above:
===================================================
"qtp1593180232-305212":
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:170)
- waiting to lock <0x0000000681692280> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
at
org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:316)
at
org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$CallbackBridge.writeSuccess(AbstractWebSocketConnection.java:86)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.notifyCallbackSuccess(FrameFlusher.java:359)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.succeedEntries(FrameFlusher.java:288)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.succeeded(FrameFlusher.java:280)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:293)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:510)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
"qtp1593180232-302665":
at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
- waiting to lock <0x0000000755589698> (a
org.apache.zeppelin.socket.NotebookSocket)
at
org.apache.zeppelin.socket.ConnectionManager.broadcast(ConnectionManager.java:242)
at
org.apache.zeppelin.socket.ConnectionManager.checkCollaborativeStatus(ConnectionManager.java:209)
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromNote(ConnectionManager.java:185)
- locked <0x0000000681692280> (a java.util.HashMap)
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:173)
- locked <0x0000000681692280> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
at
org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.remoteClose(AbstractWebSocketConnection.java:274)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:109)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:71)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
at
org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
"qtp1593180232-302348":
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:170)
- waiting to lock <0x0000000681692280> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
at
org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:225)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$Flusher.onCompleteFailure(AbstractWebSocketConnection.java:100)
at
org.eclipse.jetty.util.IteratingCallback.failed(IteratingCallback.java:402)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:155)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextOutgoingFrame(PerMessageDeflateExtension.java:123)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.access$1100(CompressExtension.java:44)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.compress(CompressExtension.java:591)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.deflate(CompressExtension.java:488)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.process(CompressExtension.java:468)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.outgoingFrame(CompressExtension.java:244)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack$Flusher.process(ExtensionStack.java:400)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:277)
at
org.eclipse.jetty.websocket.common.WebSocketSession.outgoingFrame(WebSocketSession.java:359)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.uncheckedSendFrame(WebSocketRemoteEndpoint.java:306)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendAsyncFrame(WebSocketRemoteEndpoint.java:240)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendStringByFuture(WebSocketRemoteEndpoint.java:403)
at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
- locked <0x0000000755589698> (a org.apache.zeppelin.socket.NotebookSocket)
at
org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:823)
at
org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:819)
at
org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:141)
at
org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:121)
at
org.apache.zeppelin.socket.NotebookServer.getNote(NotebookServer.java:818)
at
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:311)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
at
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
at
org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
