Jan Høydahl created SOLR-18634:
----------------------------------
Summary: Detect replicas registered in ZK that have no core on
disk at startup
Key: SOLR-18634
URL: https://issues.apache.org/jira/browse/SOLR-18634
Project: Solr
Issue Type: Improvement
Components: SolrCloud
Reporter: Jan Høydahl
h3. Problem
When a SolrCloud node starts with an empty data directory (typically a lost or
re-provisioned volume in Kubernetes), ZooKeeper still lists the replicas that
lived on that node, but core discovery finds no {{core.properties}} for them.
Today {{ZkController}} publishes all of the node's replicas as DOWN at startup,
and then registers only the cores it finds on disk. Nothing compares the two
lists. A replica that exists in ZK but not on disk stays DOWN forever, and
nothing is logged about why. The cluster keeps serving from the other replicas,
so the condition is easy to miss until someone looks at the cloud graph.
The manual fix is a CoreAdmin CREATE with the existing {{coreNodeName}}, which
the ref guide already documents as the way to "assign a new core to replace an
existing replica". The new core then does a full replication from the leader.
Many users instead resort to DELETEREPLICA + ADDREPLICA because this path is
not well known. See the operator-side discussion in
https://github.com/apache/solr-operator/issues/787.
h3. Proposal
# *Always:* after core discovery, diff the ZK replicas for this node against
the discovered cores and log a WARN per orphaned replica, naming collection,
shard, {{coreNodeName}} and whether an active leader exists elsewhere in the
shard.
# *Opt-in:* a system property / env var, e.g.
{{\-Dsolr.cloud.recreateMissingCores=true}} /
{{SOLR_CLOUD_RECREATE_MISSING_CORES=true}}, default {{false}}. When enabled,
each orphaned replica is recreated locally through the existing
{{coreNodeName}} path and left to recover from the leader.
The repair must be guarded: only recreate a core when the shard has an ACTIVE
leader on another live node. If the lost replica was the shard's only copy,
recreating it would make an empty core the leader and hide the data loss. In
that case only the WARN is emitted, and the replica stays DOWN so that an
operator can restore from backup. We could guard on free disk space too,
but it is likely better to recreate the core and let it fail replication as it
normally would on low disk.
Out of scope: partial disk loss where {{core.properties}} survives but the
index does not (see SOLR-18008), and any automatic DELETEREPLICA / ADDREPLICA.
----
🤖 Issue text authored with AI assistance, approved by me
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
