I agree and have created https://issues.apache.org/jira/browse/ZEPPELIN-1084 .
Best, Prasad On Tue, Jun 28, 2016 at 11:41 PM, Johnny W. <jzw.ser...@gmail.com> wrote: > Thanks, Prasad! This is very helpful. The version we are using already > includes Jetty 9 & Zeppelin-820. > > However, we still see web hangs quite frequently caused by broadcasting > updateNote. I may remove the synchronized block on noteSocketMap as a > temporary fix, but I am wondering whether there is a better solution. > > + zeppelin-dev, since this issue may significantly limit the scalability > of Zeppelin if there is a bad connection. One potential optimization can > be: instead of lock the whole map, use fine-grained lock on map entries. > > Best, > Johnny > > > > > On Tue, Jun 28, 2016 at 4:34 AM, Prasad Wagle <prasadwa...@gmail.com> > wrote: > >> Hi Johnny, >> >> What version of the server are you using? >> >> You may be interested in the following: >> >> Email thread >> <http://mail-archives.apache.org/mod_mbox/zeppelin-users/201604.mbox/%3CCAMKL62nmLDJRghYB39MdnbAiRT6wvso0eVePKRtKurmu5iM%2BuQ%40mail.gmail.com%3E> >> discussing zeppelin server hangs and reducing websocket connections >> From the thread: "I removed synchronized (noteSocketMap) from broadcast >> so that one bad >> socket does not hang the server." This change helps with performance >> as well and hasn't caused any problems. However, this is not a long-term >> solution and we need to find a better one. >> >> Jira issue: Reduce websocket communication by unicasting instead of >> broadcasting note list ( >> https://issues.apache.org/jira/browse/ZEPPELIN-820) >> >> Prasad >> >> On Mon, Jun 20, 2016 at 5:04 PM, Johnny W. <jzw.ser...@gmail.com> wrote: >> >>> Hi zeppelin-users, >>> >>> This is my first email to the top-level mailing list. Congratulations >>> for graduation! >>> >>> We are hitting some performance issues when multiple users are connected >>> to the Zeppelin server. From the stack trace, many of the connections are >>> blocked on a HashMap, which is locked by >>> org.apache.zeppelin.socket.NotebookServer.broadcastNote. >>> >>> Our largest notebook is around 800K, and there are around 10 - 20 >>> connections to the Zeppelin server. I think it should be we are >>> broadcasting some large amount of data to multiple users, and some slow >>> connections hang the whole web interface. >>> >>> Is there anyway to reduce the number of broadcasts to improve the web >>> performance? It is fine for us to refresh and get updates. I've attached >>> the full stack trace of this issue as well. >>> >>> Thanks! >>> >>> Johnny >>> >>> Blocking Thread: >>> -- >>> "qtp1874598090-2478" prio=10 tid=0x00007f2fb0003800 nid=0x3373 waiting >>> on condition [0x00007f329ebe9000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x0000000704e15db0> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at >>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) >>> at >>> org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) >>> at >>> org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:83) >>> at >>> org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107) >>> at >>> org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:387) >>> at >>> org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:69) >>> at >>> org.apache.zeppelin.socket.NotebookServer.broadcast(NotebookServer.java:304) >>> - locked <0x00000007006b6100> (a java.util.HashMap) >>> at >>> org.apache.zeppelin.socket.NotebookServer.broadcastNote(NotebookServer.java:384) >>> at >>> org.apache.zeppelin.socket.NotebookServer.updateNote(NotebookServer.java:492) >>> at >>> org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:181) >>> at >>> org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56) >>> at >>> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128) >>> at >>> org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) >>> at >>> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) >>> at >>> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122) >>> at >>> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161) >>> at >>> org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) >>> at >>> org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214) >>> at >>> org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) >>> at >>> org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) >>> at >>> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) >>> at >>> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) >>> at >>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) >>> at >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) >>> at >>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Blocked Thread: >>> -- >>> "qtp1874598090-2498" prio=10 tid=0x00007f306000f800 nid=0x4075 waiting >>> for monitor entry [0x00007f329eae9000] >>> java.lang.Thread.State: BLOCKED (on object monitor) >>> at >>> org.apache.zeppelin.socket.NotebookServer.addConnectionToNote(NotebookServer.java:229) >>> - waiting to lock <0x00000007006b6100> (a java.util.HashMap) >>> at >>> org.apache.zeppelin.socket.NotebookServer.sendNote(NotebookServer.java:432) >>> at >>> org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:145) >>> at >>> org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56) >>> at >>> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128) >>> at >>> org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) >>> at >>> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) >>> at >>> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122) >>> at >>> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161) >>> at >>> org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) >>> at >>> org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214) >>> at >>> org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) >>> at >>> org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) >>> at >>> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) >>> at >>> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) >>> at >>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) >>> at >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) >>> at >>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >> >
