[
https://issues.apache.org/jira/browse/HDDS-16273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16273:
----------------------------------
Labels: pull-request-available (was: )
> OM bootstrap never completes when a single SST file is larger than
> ozone.om.ratis.snapshot.max.total.sst.size
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16273
> URL: https://issues.apache.org/jira/browse/HDDS-16273
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: YUNG_CHENG
> Assignee: YUNG_CHENG
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> When a follower OM bootstraps or installs a Ratis snapshot, a single SST file
> whose size is greater than or equal to
> ozone.om.ratis.snapshot.max.total.sst.size can never be transferred, and the
> follower retries forever without any error.
> h2. Code path
> In OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir() the size budget
> is checked *before*
> the file is recorded and *before* it is added to sstFilesToExclude:
> long fileSize = Files.size(dbFile);
> if (maxTotalSstSize.get() - fileSize <= 0)
> { return false; }
> bytesRecorded += omdbArchiver.recordFileEntry(dbFile.toFile(), fileId);
> ...
> sstFilesToExclude.add(fileId);
> maxTotalSstSize is rebuilt at its full configured value on every HTTP
> request, so a file with
> size >= budget fails the check in every round. The false return propagates to
> shouldContinue, which skips omdbArchiver.setCompleted(true), so the tarball
> carries neither the file nor the completion marker.
> OMDBArchiver.writeToArchive() does not throw on an empty file map and the
> servlet still returns HTTP 200.
> On the follower, RDBSnapshotProvider.downloadDBSnapshotFromLeader() is an
> unbounded while(true) whose only exit is ratisSnapshotComplete(). The
> exclusion list for the next request is rebuilt from
> HAUtils.getExistingFiles(candidateDir), which is unchanged, so the leader
> makes the same zero-progress decision indefinitely. No error, no bound, no
> progress. The wedged thread also holds the synchronized OzoneManager monitor
> via installSnapshotFromLeader().
> Note the comparison is "<= 0", so a file exactly equal to the budget also
> stalls.
> h2. Regression
> The legacy servlet does not have this bug. OMDBCheckpointServlet.processDir()
> calls processFile()
> first and evaluates the budget afterwards, so it overshoots by one file and
> always makes progress.
> The check-before-record ordering was introduced with the inode-based transfer
> in HDDS-12984 (commit 96390ac142), which became the default in HDDS-14721
> (ozone.om.db.checkpoint.use.inode.based.transfer=true).
> HDDS-14403 raised the default budget from 100 MB to 10 GiB, which masks the
> bug but does not fix it.
> h2. Impact
> OM follower bootstrap / install-snapshot never completes. Reachable whenever
> ozone.om.ratis.snapshot.max.total.sst.size is tuned down, or when a snapshot
> SST exceeds the
> configured value. Only the snapshot-data phases are bounded (snapshot dirs,
> SST backup dir,
> compaction log dir); the active om.db phase is unbounded.
> h2. Existing coverage
> TestOMRatisSnapshotTransfer sets the limit to the *sum* of SST sizes halved,
> which by
> construction never produces a single file larger than the whole budget, so
> the failing case is untested.
> h2. Proposed fix
> Only enforce the budget once the current request has already collected at
> least one file, so every round transfers at least one file and the transfer
> always makes progress. This restores the legacy overshoot-by-one behaviour.
> Additionally, bound the follower loop so a zero-progress round fails with a
> diagnosable IOException instead of spinning forever.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
