KylinJN created ZEPPELIN-4491:
---------------------------------
Summary: Deadlock happens in Zeppelin when deleting multiple
paragraphs synchronously via RestAPI
Key: ZEPPELIN-4491
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4491
Project: Zeppelin
Issue Type: Bug
Components: Core
Affects Versions: 0.8.0
Reporter: KylinJN
Fix For: 0.8.0
The zeppelin server is consistently getting deadlocked when delete multiple
paragraphs which belong to different notes synchronously
The RestApi called is below:
|
// zeppelin-server\src\main\java\org\apache\zeppelin\rest\NotebookRestApi.java
/**
* Delete paragraph REST API
*
* @param noteId ID of Note
* @return JSON with status.OK
* @throws IOException
*/
@DELETE
@Path("\{noteId}/paragraph/\{paragraphId}")
@ZeppelinApi
public Response deleteParagraph(@PathParam("noteId") String noteId,
@PathParam("paragraphId") String paragraphId) throws IOException {
LOG.info("delete paragraph {} {}", noteId, paragraphId);
Note note = notebook.getNote(noteId);
checkIfNoteIsNotNull(note);
checkIfUserCanRead(noteId,
"Insufficient privileges you cannot remove paragraph from this note");
Paragraph p = note.getParagraph(paragraphId);
checkIfParagraphIsNotNull(p);
AuthenticationInfo subject = new
AuthenticationInfo(SecurityUtils.getPrincipal());
note.removeParagraph(SecurityUtils.getPrincipal(), paragraphId);
note.persist(subject);
notebookServer.broadcastNote(note);
return new JsonResponse(Status.OK, "").build();
}
|
The relevant thread dump is below:
|
Server compiler detected.
JVM version is 25.212-b10
Deadlock Detection:
Found one Java-level deadlock:
=============================
"qtp1188392295-14051":
waiting to lock Monitor@0x00007f3dc800a108 (Object@0x00000000c7706788, a
java/util/LinkedList),
which is held by "qtp1188392295-14087"
"qtp1188392295-14087":
waiting to lock Monitor@0x00007f3d10009278 (Object@0x00000000c78d3e78, a
java/util/LinkedList),
which is held by "qtp1188392295-14051"
Found a total of 1 deadlock.
|
The stacktrace information is below:
|
“qtp1188392295-14087”：
[1] org.apache.zeppelin.notebook.Note.getParagraphs (Note.java:750)
[2] org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime
(Notebook.java:871)
[3] org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo
(NotebookServer.java:554)
[4]
org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove
(NotebookServer.java:2,180)
[5] org.apache.zeppelin.notebook.Notebook.onParagraphRemove
(Notebook.java:1,083)
[6] org.apache.zeppelin.notebook.Note.removeParagraph (Note.java:446)
[7] org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph
(NotebookRestApi.java:615)
[8] sun.reflect.GeneratedMethodAccessor27.invoke (null)
[9] sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
[10] java.lang.reflect.Method.invoke (Method.java:498)
[11]
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0
(ResourceMethodInvocationHandlerFactory.java:76)
[12]
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$159.invoke
(null)
[13]
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run
(AbstractJavaResourceMethodDispatcher.java:148)
[14]
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke
(AbstractJavaResourceMethodDispatcher.java:191)
[15]
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch
(JavaResourceMethodDispatcherProvider.java:200)
[16]
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch
(AbstractJavaResourceMethodDispatcher.java:103)
[17] org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke
(ResourceMethodInvoker.java:493)
[18] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply
(ResourceMethodInvoker.java:415)
[19] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply
(ResourceMethodInvoker.java:104)
[20] org.glassfish.jersey.server.ServerRuntime$1.run (ServerRuntime.java:277)
[21] org.glassfish.jersey.internal.Errors$1.call (Errors.java:272)
[22] org.glassfish.jersey.internal.Errors$1.call (Errors.java:268)
[23] org.glassfish.jersey.internal.Errors.process (Errors.java:316)
[24] org.glassfish.jersey.internal.Errors.process (Errors.java:298)
[25] org.glassfish.jersey.internal.Errors.process (Errors.java:268)
[26] org.glassfish.jersey.process.internal.RequestScope.runInScope
(RequestScope.java:289)
[27] org.glassfish.jersey.server.ServerRuntime.process (ServerRuntime.java:256)
[28] org.glassfish.jersey.server.ApplicationHandler.handle
(ApplicationHandler.java:703)
[29] org.glassfish.jersey.servlet.WebComponent.serviceImpl
(WebComponent.java:416)
[30] org.glassfish.jersey.servlet.WebComponent.service (WebComponent.java:370)
[31] org.glassfish.jersey.servlet.ServletContainer.service
(ServletContainer.java:389)
[32] org.glassfish.jersey.servlet.ServletContainer.service
(ServletContainer.java:342)
[33] org.glassfish.jersey.servlet.ServletContainer.service
(ServletContainer.java:229)
[34] org.eclipse.jetty.servlet.ServletHolder.handle (ServletHolder.java:865)
[35] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1,655)
[36] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter
(ProxiedFilterChain.java:61)
[37] org.apache.shiro.web.servlet.AdviceFilter.executeChain
(AdviceFilter.java:108)
[38] org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal
(AdviceFilter.java:137)
[39] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter
(OncePerRequestFilter.java:125)
[40] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter
(ProxiedFilterChain.java:66)
[41] org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain
(AbstractShiroFilter.java:449)
[42] org.apache.shiro.web.servlet.AbstractShiroFilter$1.call
(AbstractShiroFilter.java:365)
[43] org.apache.shiro.subject.support.SubjectCallable.doCall
(SubjectCallable.java:90)
[44] org.apache.shiro.subject.support.SubjectCallable.call
(SubjectCallable.java:83)
[45] org.apache.shiro.subject.support.DelegatingSubject.execute
(DelegatingSubject.java:383)
[46] org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal
(AbstractShiroFilter.java:362)
[47] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter
(OncePerRequestFilter.java:125)
[48] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1,642)
[49] org.apache.zeppelin.server.CorsFilter.doFilter (CorsFilter.java:72)
[50] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1,642)
[51] org.eclipse.jetty.servlet.ServletHandler.doHandle
(ServletHandler.java:533)
[52] org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:146)
[53] org.eclipse.jetty.security.SecurityHandler.handle
(SecurityHandler.java:548)
[54] org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
[55] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:257)
[56] org.eclipse.jetty.server.session.SessionHandler.doHandle
(SessionHandler.java:1,595)
[57] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:255)
[58] org.eclipse.jetty.server.handler.ContextHandler.doHandle
(ContextHandler.java:1,317)
[59] org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:203)
[60] org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473)
[61] org.eclipse.jetty.server.session.SessionHandler.doScope
(SessionHandler.java:1,564)
[62] org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:201)
[63] org.eclipse.jetty.server.handler.ContextHandler.doScope
(ContextHandler.java:1,219)
[64] org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:144)
[65] org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
(ContextHandlerCollection.java:219)
[66] org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
[67] org.eclipse.jetty.server.Server.handle (Server.java:531)
[68] org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
[69] org.eclipse.jetty.server.HttpConnection.onFillable
(HttpConnection.java:260)
[70] org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
(AbstractConnection.java:281)
[71] org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
[72] org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
[73] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
(EatWhatYouKill.java:333)
[74] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:310)
[75] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
[76] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
[77] org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
[78] org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
[79] org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
[80] java.lang.Thread.run (Thread.java:748)
------------------------------------------------------------------------------------------------
"qtp1188392295-14051":
[1] org.apache.zeppelin.notebook.Note.getParagraphs (Note.java:750)
[2] org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime
(Notebook.java:871)
[3] org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo
(NotebookServer.java:554)
[4]
org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove
(NotebookServer.java:2,180)
[5] org.apache.zeppelin.notebook.Notebook.onParagraphRemove
(Notebook.java:1,083)
[6] org.apache.zeppelin.notebook.Note.removeParagraph (Note.java:446)
[7] org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph
(NotebookRestApi.java:615)
[8] sun.reflect.GeneratedMethodAccessor27.invoke (null)
[9] sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
[10] java.lang.reflect.Method.invoke (Method.java:498)
[11]
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0
(ResourceMethodInvocationHandlerFactory.java:76)
[12]
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$159.invoke
(null)
[13]
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run
(AbstractJavaResourceMethodDispatcher.java:148)
[14]
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke
(AbstractJavaResourceMethodDispatcher.java:191)
[15]
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch
(JavaResourceMethodDispatcherProvider.java:200)
[16]
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch
(AbstractJavaResourceMethodDispatcher.java:103)
[17] org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke
(ResourceMethodInvoker.java:493)
[18] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply
(ResourceMethodInvoker.java:415)
[19] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply
(ResourceMethodInvoker.java:104)
[20] org.glassfish.jersey.server.ServerRuntime$1.run (ServerRuntime.java:277)
[21] org.glassfish.jersey.internal.Errors$1.call (Errors.java:272)
[22] org.glassfish.jersey.internal.Errors$1.call (Errors.java:268)
[23] org.glassfish.jersey.internal.Errors.process (Errors.java:316)
[24] org.glassfish.jersey.internal.Errors.process (Errors.java:298)
[25] org.glassfish.jersey.internal.Errors.process (Errors.java:268)
[26] org.glassfish.jersey.process.internal.RequestScope.runInScope
(RequestScope.java:289)
[27] org.glassfish.jersey.server.ServerRuntime.process (ServerRuntime.java:256)
[28] org.glassfish.jersey.server.ApplicationHandler.handle
(ApplicationHandler.java:703)
[29] org.glassfish.jersey.servlet.WebComponent.serviceImpl
(WebComponent.java:416)
[30] org.glassfish.jersey.servlet.WebComponent.service (WebComponent.java:370)
[31] org.glassfish.jersey.servlet.ServletContainer.service
(ServletContainer.java:389)
[32] org.glassfish.jersey.servlet.ServletContainer.service
(ServletContainer.java:342)
[33] org.glassfish.jersey.servlet.ServletContainer.service
(ServletContainer.java:229)
[34] org.eclipse.jetty.servlet.ServletHolder.handle (ServletHolder.java:865)
[35] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1,655)
[36] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter
(ProxiedFilterChain.java:61)
[37] org.apache.shiro.web.servlet.AdviceFilter.executeChain
(AdviceFilter.java:108)
[38] org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal
(AdviceFilter.java:137)
[39] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter
(OncePerRequestFilter.java:125)
[40] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter
(ProxiedFilterChain.java:66)
[41] org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain
(AbstractShiroFilter.java:449)
[42] org.apache.shiro.web.servlet.AbstractShiroFilter$1.call
(AbstractShiroFilter.java:365)
[43] org.apache.shiro.subject.support.SubjectCallable.doCall
(SubjectCallable.java:90)
[44] org.apache.shiro.subject.support.SubjectCallable.call
(SubjectCallable.java:83)
[45] org.apache.shiro.subject.support.DelegatingSubject.execute
(DelegatingSubject.java:383)
[46] org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal
(AbstractShiroFilter.java:362)
[47] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter
(OncePerRequestFilter.java:125)
[48] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1,642)
[49] org.apache.zeppelin.server.CorsFilter.doFilter (CorsFilter.java:72)
[50] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1,642)
[51] org.eclipse.jetty.servlet.ServletHandler.doHandle
(ServletHandler.java:533)
[52] org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:146)
[53] org.eclipse.jetty.security.SecurityHandler.handle
(SecurityHandler.java:548)
[54] org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
[55] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:257)
[56] org.eclipse.jetty.server.session.SessionHandler.doHandle
(SessionHandler.java:1,595)
[57] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:255)
[58] org.eclipse.jetty.server.handler.ContextHandler.doHandle
(ContextHandler.java:1,317)
[59] org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:203)
[60] org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473)
[61] org.eclipse.jetty.server.session.SessionHandler.doScope
(SessionHandler.java:1,564)
[62] org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:201)
[63] org.eclipse.jetty.server.handler.ContextHandler.doScope
(ContextHandler.java:1,219)
[64] org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:144)
[65] org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
(ContextHandlerCollection.java:219)
[66] org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
[67] org.eclipse.jetty.server.Server.handle (Server.java:531)
[68] org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
[69] org.eclipse.jetty.server.HttpConnection.onFillable
(HttpConnection.java:260)
[70] org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
(AbstractConnection.java:281)
[71] org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
[72] org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
[73] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
(EatWhatYouKill.java:333)
[74] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:310)
[75] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
[76] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
[77] org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
[78] org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
[79] org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
[80] java.lang.Thread.run (Thread.java:748)
|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
