[
https://issues.apache.org/jira/browse/SOLR-15371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363166#comment-17363166
]
Jason Gerlowski commented on SOLR-15371:
----------------------------------------
There may well be a bug you're hitting, but I don't think it could be a
race-condition between different parts of Solr to create/read directories. As
far as I recall, the backup code creates directories on the node receiving the
API call and on the "overseer" node that kicks off the lion's share of the
work. These two steps occur serially, and before the overseer fans out
requests to backup each individual cores. Weird stuff happens all the time:
never say never and all that...but it's hard to imagine the bug being a
read/write race condition, since dir-creation happens synchronously and in the
same thread that later sends the core-level requests.
Side note: I assume that you have verified that the NFS is present on *all*
Solr nodes, and that there's no network issues that'd cause it to appear
temporarily unavailable from Solr's perspective? Either of those are easy to
imagine causing the error you mentioned.
> Backups randomly fail sometimes
> -------------------------------
>
> Key: SOLR-15371
> URL: https://issues.apache.org/jira/browse/SOLR-15371
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
> Affects Versions: 8.5.2, 8.8.2
> Reporter: Roy Perkins
> Priority: Major
>
> Hi, we have an issue where sometimes one shard fails to backup due to what
> might be a race condition in creating the folder/starting the backup. When
> this happens, we have to restart the first server in a shard to get the
> backup to succeed again. The cluster backs up to a shared NFS mount. 4/5
> times the backup goes fine without issues (there is even another collection
> that the backup will run for later in the morning that will succeed fine even
> though it's all the same servers) Below is the error I get.
> {code:java}
> "Response":"Failed to backup core=slprod_shard4_replica_n6 because
> org.apache.solr.common.SolrException: Directory to contain snapshots doesn't
> exist: file:///mnt/solr_backups/slprod/slprod-04-25-2021. Note that
> Backup/Restore of a SolrCloud collection requires a shared file system
> mounted at the same path on all nodes!"},
> {code}
> And below is the line I use to backup with (obviously with bash variables set
> earlier in the script)
> {code:java}
> curl -s
> "http://localhost:8983/solr/admin/collections?action=BACKUP&name=${COLLECTION}-${DATE}&collection=${COLLECTION}&location=${BACKUP_PATH}&async=1000";
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
