[
https://issues.apache.org/jira/browse/SOLR-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson updated SOLR-7836:
---------------------------------
Attachment: SOLR-7836.patch
Looking for comments:
This looks like I changed more than I did. Expanded the scope of the
try/finally block in newIndexWriter, moved the common code for initializing
outside an else clause and put a try/finally block in closeIndexWriter. The
rest of the diff is just noise due to indentation.
I have a field report and stack traces of a deadlock, here's the stack trace:
I think that the first two in DefaultSolrCoreState are where I'm guessing the
root of the problem lies.
********************First thread in DefaultSolrCoreState
- The thread that owns the updateLock is stuck waiting for pauseWriter to go to
false so it can get the IndexWriter:
java.lang.Object@c0d4b9
java.lang.Object.wait(Native Method)
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:94)
org.apache.solr.update.DirectUpdateHandler2.addAndDelete(DirectUpdateHandler2.java:436)
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:216)
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:928)
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082)
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:695)
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
org.apache.solr.core.SolrCore.execute(SolrCore.java:1956)
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:799)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:422)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
com.apple.cie.search.plugin.auth.TrustFilter.doFilter(TrustFilter.java:43)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
org.eclipse.jetty.server.Server.handle(Server.java:368)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
java.lang.Thread.run(Thread.java:745)
********************Second thread in DefaultSolrCoreState
- The thread that sets pauseWriter to true that is supposed to later set it to
false is stuck on waiting for writerFree to go to true:
java.lang.Object@c0d4b9
java.lang.Object.wait(Native Method)
org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:156)
org.apache.solr.core.SolrCore.reload(SolrCore.java:431)
org.apache.solr.core.CoreContainer.reload(CoreContainer.java:586)
org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:701)
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:225)
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:188)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:751)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:259)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
com.apple.cie.search.plugin.auth.TrustFilter.doFilter(TrustFilter.java:43)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
org.eclipse.jetty.server.Server.handle(Server.java:368)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
java.lang.Thread.run(Thread.java:745)
****************Lots of threads stuck in DirectUpdateHandler2, but I suspect
these aren't where the problem really is:
java.lang.Object@73cdbe11
org.apache.solr.update.DirectUpdateHandler2.addAndDelete(DirectUpdateHandler2.java:435)
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:216)
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:928)
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082)
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:695)
> Possible deadlock when closing refcounted index writers.
> --------------------------------------------------------
>
> Key: SOLR-7836
> URL: https://issues.apache.org/jira/browse/SOLR-7836
> Project: Solr
> Issue Type: Bug
> Reporter: Erick Erickson
> Attachments: SOLR-7836.patch
>
>
> Preliminary patch for what looks like a possible race condition between
> writerFree and pauseWriter in DefaultSorlCoreState.
> Looking for comments and/or why I'm completely missing the boat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
