[
https://issues.apache.org/jira/browse/SOLR-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141166#comment-18141166
]
Andrzej Bialecki commented on SOLR-18634:
-----------------------------------------
{{solr.cloud.delete.unknown.cores.enabled}} doesn't cover the same scenario as
described here, it removes local data of a replica that is no longer in
ClusterState. The setting proposed here does something almost but not quite
opposite ;) it optionally back-fills the replica data if it's locally missing,
or cleans up stale replica state in ClusterState that points to a locally
missing data.
> Detect replicas registered in ZK that have no core on disk at startup
> ---------------------------------------------------------------------
>
> Key: SOLR-18634
> URL: https://issues.apache.org/jira/browse/SOLR-18634
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Reporter: Jan Høydahl
> Priority: Major
>
> h3. Problem
> When a SolrCloud node starts with an empty data directory (typically a lost
> or re-provisioned volume in Kubernetes), ZooKeeper still lists the replicas
> that lived on that node, but core discovery finds no {{core.properties}} for
> them.
> Today {{ZkController}} publishes all of the node's replicas as DOWN at
> startup, and then registers only the cores it finds on disk. Nothing compares
> the two lists. A replica that exists in ZK but not on disk stays DOWN
> forever, and nothing is logged about why. The cluster keeps serving from the
> other replicas, so the condition is easy to miss until someone looks at the
> cloud graph.
> The manual fix is a CoreAdmin CREATE with the existing {{coreNodeName}},
> which the ref guide already documents as the way to "assign a new core to
> replace an existing replica". The new core then does a full replication from
> the leader. Many users instead resort to DELETEREPLICA + ADDREPLICA because
> this path is not well known. See the operator-side discussion in
> https://github.com/apache/solr-operator/issues/787.
> h3. Proposal
> # *Always:* after core discovery, diff the ZK replicas for this node against
> the discovered cores and log a WARN per orphaned replica, naming collection,
> shard, {{coreNodeName}} and whether an active leader exists elsewhere in the
> shard.
> # *Opt-in:* a system property / env var, e.g.
> {{\-Dsolr.cloud.recreateMissingCores=true}} /
> {{SOLR_CLOUD_RECREATE_MISSING_CORES=true}}, default {{false}}. When enabled,
> each orphaned replica is recreated locally through the existing
> {{coreNodeName}} path and left to recover from the leader.
> The repair must be guarded: only recreate a core when the shard has an ACTIVE
> leader on another live node. If the lost replica was the shard's only copy,
> recreating it would make an empty core the leader and hide the data loss. In
> that case only the WARN is emitted, and the replica stays DOWN so that an
> operator can restore from backup. We could guard on free disk space too,
> but it is likely better to recreate the core and let it fail replication as
> it normally would on low disk.
> Out of scope: partial disk loss where {{core.properties}} survives but the
> index does not (see SOLR-18008), and any automatic DELETEREPLICA / ADDREPLICA.
> ----
> 🤖 Issue text authored with AI assistance, approved by me
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
