Siyao Meng created HDDS-16864:
---------------------------------
Summary: OM follower leaves the downloaded checkpoint tarball in
the ratis snapshot dir when untar fails
Key: HDDS-16864
URL: https://issues.apache.org/jira/browse/HDDS-16864
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: TestUntarFailureLeaksTarball.java,
delete-checkpoint-tarball-when-untar-fails.patch
h3. Mechanism
{{RDBSnapshotProvider.untarContentsOfTarball}} calls {{FileUtil.unTar}} and
deletes the downloaded tarball only after it returns. If untar throws, the
{{om.db-<leader>-<millis>.tar}} file stays in the OM ratis snapshot dir. No
other path removes it: {{OmRatisSnapshotProvider.downloadSnapshot}} deletes the
file only when the HTTP download itself fails,
{{OzoneManager.cleanupCheckpoint}} deletes only the candidate dir,
{{RDBSnapshotProvider.init}} (OM startup, leader change,
{{notifySnapshotInstalled}}) empties only the candidate dir, and a later
successful install deletes only its own tarball. Every retry uses a new file
name, so each failed attempt leaves one more file. The code is unchanged since
HDDS-7995 (1.4.0) and is present in master.
Related: HDDS-12820, whose description assumes {{init()}} removes partial
downloads; it removes only the candidate dir.
h3. Trigger
# A follower OM downloads a checkpoint part from the leader.
# The download completes but the tarball cannot be extracted. Two ways, both
argued from the code and not run end to end:
{{OMDBCheckpointServletInodeBasedXfer.processMetadataSnapshotRequest}} catches
an {{IOException}} from {{OMDBArchiver.writeToArchive}} after the response is
committed, logs it and returns normally (its {{setStatus(500)}} has no effect
at that point), so the follower gets HTTP 200 with a truncated tar; or
extraction runs out of space on the snapshot volume.
# {{unTar}} throws, {{installSnapshotFromLeader}} cleans the candidate dir and
returns null, and the leader retries the install.
h3. Impact
* Each failed untar permanently consumes the size of the failing part on the OM
ratis snapshot volume. Restart does not remove it.
* Since HDDS-15171 (2.3.0, and master), on a tight disk the leftover file makes
the next attempts fail the bootstrap disk space check before downloading, so
the follower cannot install a checkpoint until an operator deletes the file by
hand. The error points at disk space but not at the leftover tarball. Earlier
releases have no such check, so the next download runs into the full disk while
writing instead (argued from the code, not run).
* If the disk check still passes, every failed retry adds another tarball.
* Recon uses the same {{RDBSnapshotProvider}} code (argued, not run for Recon).
h3. Reproduction
Deterministic, unmodified source. [^TestUntarFailureLeaksTarball.java]
({{hadoop-ozone/ozone-manager}}) calls
{{OmRatisSnapshotProvider.downloadDBSnapshotFromLeader}} over its real download
path with only {{HttpURLConnection}} mocked, as in
{{TestOmRatisSnapshotProvider}}: the leader answers HTTP 200 with a tar that
ends in the middle of an entry. After each of three failed attempts the test
performs the follower's own cleanup (delete the candidate dir, then
{{init()}}). The test asserts that no tarball is left, so it fails while the
defect is present: on the unfixed code three {{om.db-om1-*.tar}} files remain,
one per attempt. It passes with the patch.
h3. Patch
[^delete-checkpoint-tarball-when-untar-fails.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
The existing {{FileUtil.fullyDelete(snapshot)}} moves into a {{finally}} around
{{FileUtil.unTar}}, so the tarball is deleted whether or not extraction
succeeds and the exception still propagates. The success path is unchanged.
Tarballs left by earlier failures or by a process crash between download and
untar are not removed by the patch.
Covered by {{testDownloadedTarballDeletedWhenUntarFails}} in the existing
{{TestRDBSnapshotProvider}}: a truncated first part must fail without leaving a
tarball, and the retry with a complete part must return the checkpoint without
leaving one. Without the change it fails with the tarball still present. With
it {{TestRDBSnapshotProvider}}, {{TestRDBStore}},
{{TestOmRatisSnapshotProvider}} and {{TestReconRDBSnapshotProvider}} (34 tests)
pass and checkstyle is clean.
Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
