[
https://issues.apache.org/jira/browse/SOLR-18584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18584:
----------------------------------
Labels: pull-request-available (was: )
> A failed RESTORE to a new collection leaves its shards in CONSTRUCTION, and
> retrying the restore doesn't fix them
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18584
> URL: https://issues.apache.org/jira/browse/SOLR-18584
> Project: Solr
> Issue Type: Bug
> Components: Backup/Restore
> Reporter: Boyi Dai
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When RESTORE targets a collection that doesn't exist, RestoreCmd
> (RestoreOnANewCollection) creates the collection, marks all its shards
> CONSTRUCTION, creates one replica per shard, installs the backed up data on
> each shard, and only then marks the shards ACTIVE. If installing the data
> fails for a shard, requestShardsToRestore() throws and
> markAllShardsAsActive() is never called. The collection is left in ZooKeeper
> with every shard in CONSTRUCTION.
> Unlike a failure to create the initial replicas, which deletes the
> collection, nothing cleans this up. Retrying the same RESTORE now finds the
> collection, so it takes the existing collection path
> (RestoreOnExistingCollection). That path restores the data and reports
> success, but never marks CONSTRUCTION shards ACTIVE. Updates to the
> collection are then rejected because no active shard covers their hash range.
> This was found through code analysis, and I reproduced it with a test using
> the test framework's error-throwing backup repository. It happens when
> RESTORE is called synchronously. An async RESTORE doesn't abort when
> restoring a shard fails (see the TODO in
> ShardRequestTracker.processResponses): it reports FAILED, but still marks the
> shards ACTIVE and leaves the collection in place without the data that failed
> to restore.
> *Steps to reproduce:*
> # Back up a collection.
> # Synchronously RESTORE the backup to a new collection T, with restoring the
> shard data failing (e.g. the backup repository fails to copy an index file).
> # Retry the same RESTORE once the repository works again.
> # Result: the retry succeeds, but T's shards stay in CONSTRUCTION, so updates
> to T fail.
> *Expected:* a failed restore to a new collection shouldn't leave a
> half-restored collection behind, and a retry should produce a usable
> collection.
> *Proposed fix:* if anything fails after the collection is created and before
> its shards are marked ACTIVE, delete the collection, as is already done when
> the initial replicas can't be created, and treat a shard restore failure
> recorded by an async request the same way. A retry then restores to a new
> collection again. I have a patch with a test and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
