zhaoming.chen created ZEPPELIN-5643:
---------------------------------------
Summary: deadlock in getNote
Key: ZEPPELIN-5643
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5643
Project: Zeppelin
Issue Type: Bug
Components: zeppelin-server
Affects Versions: 0.10.0
Environment: 0.10.0
Reporter: zhaoming.chen
I can login the zeppelin, but can't enter note. the stack info of
zeppelin-server process like this:
Found one Java-level deadlock:
=============================
"qtp1593180232-1490":
waiting to lock monitor 0x00007fcba40055b8 (object 0x0000000681523a68, a
java.util.HashMap),
which is held by "qtp1593180232-819"
"qtp1593180232-819":
waiting to lock monitor 0x00007fcba4005668 (object 0x00000006a2988898, a
org.apache.zeppelin.socket.NotebookSocket),
which is held by "qtp1593180232-795"
"qtp1593180232-795":
waiting to lock monitor 0x00007fcba40055b8 (object 0x0000000681523a68, a
java.util.HashMap),
which is held by "qtp1593180232-819"
Java stack information for the threads listed above:
"qtp1593180232-1490" #1490 prio=5 os_prio=0 tid=0x00007fcadc013000 nid=0x45f5
waiting for monitor entry [0x00007fca956c9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:171)
- waiting to lock <0x0000000681523a68> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer$9.onSuccess(NotebookServer.java:888)
at
org.apache.zeppelin.socket.NotebookServer$9.onSuccess(NotebookServer.java:879)
at
org.apache.zeppelin.service.NotebookService.getHomeNote(NotebookService.java:114)
at
org.apache.zeppelin.socket.NotebookServer.getHomeNote(NotebookServer.java:878)
at
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:308)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
at
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
at
org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
"qtp1593180232-819" #819 prio=5 os_prio=0 tid=0x00007fcb3c003800 nid=0x3967
waiting for monitor entry [0x00007fcb94fd0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
- waiting to lock <0x00000006a2988898> (a
org.apache.zeppelin.socket.NotebookSocket)
at
org.apache.zeppelin.socket.ConnectionManager.broadcast(ConnectionManager.java:242)
at
org.apache.zeppelin.socket.ConnectionManager.checkCollaborativeStatus(ConnectionManager.java:209)
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromNote(ConnectionManager.java:185)
- locked <0x0000000681523a68> (a java.util.HashMap)
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:173)
- locked <0x0000000681523a68> (a java.util.HashMap))
at
org.apache.zeppelin.socket.ConnectionManager.addNoteConnection(ConnectionManager.java:107)
- locked <0x0000000681523a68> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:822)
at
org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:819)
at
org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:141)
at
org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:121)
at
org.apache.zeppelin.socket.NotebookServer.getNote(NotebookServer.java:818)
at
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:311)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
at
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
at
org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
"qtp1593180232-795" #795 prio=5 os_prio=0 tid=0x00007fcae00ef000 nid=0x3944
waiting for monitor entry [0x00007fcb958d2000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:171)
- waiting to lock <0x0000000681523a68> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
at
org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:225)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$Flusher.onCompleteFailure(AbstractWebSocketConnection.java:100)
at
org.eclipse.jetty.util.IteratingCallback.failed(IteratingCallback.java:402)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
at
org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:155)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextOutgoingFrame(PerMessageDeflateExtension.java:123)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.access$1100(CompressExtension.java:44)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.compress(CompressExtension.java:591)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.deflate(CompressExtension.java:488)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.process(CompressExtension.java:468)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.outgoingFrame(CompressExtension.java:244)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack$Flusher.process(ExtensionStack.java:400)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:277)
at
org.eclipse.jetty.websocket.common.WebSocketSession.outgoingFrame(WebSocketSession.java:359)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.uncheckedSendFrame(WebSocketRemoteEndpoint.java:306)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendAsyncFrame(WebSocketRemoteEndpoint.java:240)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendStringByFuture(WebSocketRemoteEndpoint.java:403)
at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
- locked <0x00000006a2988898> (a org.apache.zeppelin.socket.NotebookSocket)
at
org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:823)
at
org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:819)
at
org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:141)
at
org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:121)
at
org.apache.zeppelin.socket.NotebookServer.getNote(NotebookServer.java:818)
at
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:311)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
at
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
at
org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
at
org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
at
org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
