[
https://issues.apache.org/jira/browse/SOLR-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131122#comment-18131122
]
Nick Shanin commented on SOLR-6213:
-----------------------------------
🤖 _AI text below_ 🤖 _(reviewed by Nick Shanin)_
Two PRs are open against this ticket because the tracing turned up two separate
defects behind the election stalls:
* [PR #5209|https://github.com/apache/solr/pull/5209] caps the failed watch
retries in {{LeaderElector}}: when watch registration keeps failing, the retry
loop has no cap.
* [PR #5210|https://github.com/apache/solr/pull/5210] runs leader election
rejoins on their own thread, so a blocked rejoin no longer stalls the election
loop.
Each fix is independent, and either can be reviewed and merged on its own.
> StackOverflowException in Solr cloud's leader election
> ------------------------------------------------------
>
> Key: SOLR-6213
> URL: https://issues.apache.org/jira/browse/SOLR-6213
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10, 6.0
> Reporter: Dawid Weiss
> Priority: Critical
> Labels: pull-request-available
> Attachments: stackoverflow.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This is what's causing test hangs (at least on FreeBSD, LUCENE-5786),
> possibly on other machines too. The problem is stack overflow from looped
> calls in:
> {code}
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> >
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> >
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
> > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> >
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> {code}
> These routines attempt to log information to loggers, which in turn attempts
> to serialize messages back to the master (test process). When the stack is
> exhausted the serialization process fails and breaks the communication with
> the master test node.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
