Hi Johnny, What version of the server are you using?
You may be interested in the following: Email thread <http://mail-archives.apache.org/mod_mbox/zeppelin-users/201604.mbox/%3CCAMKL62nmLDJRghYB39MdnbAiRT6wvso0eVePKRtKurmu5iM%2BuQ%40mail.gmail.com%3E> discussing zeppelin server hangs and reducing websocket connections >From the thread: "I removed synchronized (noteSocketMap) from broadcast so that one bad socket does not hang the server." This change helps with performance as well and hasn't caused any problems. However, this is not a long-term solution and we need to find a better one. Jira issue: Reduce websocket communication by unicasting instead of broadcasting note list (https://issues.apache.org/jira/browse/ZEPPELIN-820) Prasad On Mon, Jun 20, 2016 at 5:04 PM, Johnny W. <jzw.ser...@gmail.com> wrote: > Hi zeppelin-users, > > This is my first email to the top-level mailing list. Congratulations for > graduation! > > We are hitting some performance issues when multiple users are connected > to the Zeppelin server. From the stack trace, many of the connections are > blocked on a HashMap, which is locked by > org.apache.zeppelin.socket.NotebookServer.broadcastNote. > > Our largest notebook is around 800K, and there are around 10 - 20 > connections to the Zeppelin server. I think it should be we are > broadcasting some large amount of data to multiple users, and some slow > connections hang the whole web interface. > > Is there anyway to reduce the number of broadcasts to improve the web > performance? It is fine for us to refresh and get updates. I've attached > the full stack trace of this issue as well. > > Thanks! > > Johnny > > Blocking Thread: > -- > "qtp1874598090-2478" prio=10 tid=0x00007f2fb0003800 nid=0x3373 waiting on > condition [0x00007f329ebe9000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000704e15db0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) > at > org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:83) > at > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107) > at > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:387) > at > org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:69) > at > org.apache.zeppelin.socket.NotebookServer.broadcast(NotebookServer.java:304) > - locked <0x00000007006b6100> (a java.util.HashMap) > at > org.apache.zeppelin.socket.NotebookServer.broadcastNote(NotebookServer.java:384) > at > org.apache.zeppelin.socket.NotebookServer.updateNote(NotebookServer.java:492) > at > org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:181) > at > org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56) > at > org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128) > at > org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) > at > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) > at > org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122) > at > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161) > at > org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) > at > org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214) > at > org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) > at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) > at > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) > at > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:745) > > Blocked Thread: > -- > "qtp1874598090-2498" prio=10 tid=0x00007f306000f800 nid=0x4075 waiting for > monitor entry [0x00007f329eae9000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.zeppelin.socket.NotebookServer.addConnectionToNote(NotebookServer.java:229) > - waiting to lock <0x00000007006b6100> (a java.util.HashMap) > at > org.apache.zeppelin.socket.NotebookServer.sendNote(NotebookServer.java:432) > at > org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:145) > at > org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56) > at > org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128) > at > org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) > at > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) > at > org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122) > at > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161) > at > org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) > at > org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214) > at > org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) > at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) > at > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) > at > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) > at > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:745) > >
