[
https://issues.apache.org/jira/browse/SOLR-18569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131023#comment-18131023
]
Jason Gerlowski commented on SOLR-18569:
----------------------------------------
It looks like the "removeEphemeralLiveNode" method actually does clean up
node_role entries on "main" and "branch_10x"...it's just branch_9x missing the
fix.
(It looks like this was something Houston caught as a part of his curator work
in SOLR-16116, which couldn't be backported as far as 9.x I guess?)
So here's what changes I think we need for each branch:
* {{main}} - change ZkController.checkForExistingEphemeralNode to check for
both live_node and "node_roles" entries
* {{branch_10x}} - same as above
* {{branch_9x}} and {{branch_9_11} - same as above, *and* add the missing
cleanup to ZkController.removeEphemeralLiveNode
> Solr should clean up /node_roles entries, like /live_nodes
> ----------------------------------------------------------
>
> Key: SOLR-18569
> URL: https://issues.apache.org/jira/browse/SOLR-18569
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10.1
> Reporter: Jason Gerlowski
> Assignee: Jason Gerlowski
> Priority: Major
>
> On Solr startup, ZkController.createEphemeralLiveNode creates
> {{/live_nodes/<node>}} and one ephemeral {{/node_roles/<role>/<mode>/<node>}}
> znode per node role in a single atomic ZK "multi" operation.
> On graceful shutdown, ZkController.removeEphemeralLiveNode deletes
> {{/live_nodes/<node>}}, but never does any cleanup for the corresponding
> {{/node_roles}} entries.
> These lingering entries are ephemeral and will get removed from ZK within
> {{zkClientTimeout}} millis after Solr shuts down. But they can still cause
> problems! For instance: we periodically see problems on a Solr restart
> because the new Solr node tries to create its {{/node_role}} entries but
> fails because the nodes already exist!
> {code}
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1778)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1650)
> at
> org.apache.solr.common.cloud.SolrZkClient.lambda$multi$12(SolrZkClient.java:792)
> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
> at org.apache.solr.common.cloud.SolrZkClient.multi(SolrZkClient.java:792)
> at
> org.apache.solr.cloud.ZkController.createEphemeralLiveNode(ZkController.java:1244)
> at org.apache.solr.cloud.ZkController.init(ZkController.java:1059)
> at org.apache.solr.cloud.ZkController.<init>(ZkController.java:408)
> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:139)
> at org.apache.solr.core.CoreContainer.loadInternal(CoreContainer.java:874)
> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:808)
> {code}
> We should give ZkController.removeEphemeralLiveNode the same cleanup for
> /node_roles that it already has for /live_nodes, so that restarts don't hit
> this on startup.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
