[
https://issues.apache.org/jira/browse/HDDS-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16688:
------------------------------
Attachment: HDDS-16688-MC-3-checkpoint-prelock-sst.patch
> OM checkpoint transfers snapshot SST files before taking snapshotCache.lock()
> -----------------------------------------------------------------------------
>
> Key: HDDS-16688
> URL: https://issues.apache.org/jira/browse/HDDS-16688
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16688-MC-3-checkpoint-prelock-sst.patch
>
>
> h3. Mechanism
> In {{OMDBCheckpointServletInodeBasedXfer.collectDbDataToTransfer}} the
> snapshot directories are enumerated at {{:259}} and their SST files
> transferred by the loop at {{:277-284}}, but {{snapshotCache.lock()}} is only
> taken at {{:315-316}}. The block comment at {{:303-313}} describes precisely
> the race the loop above it is exposed to.
> The loop uses the 5 argument {{collectFilesFromDir}} overload ({{:501-510}}),
> which delegates with {{ignoreNoSuchFileException = false}} ({{:512-516}}),
> while the sibling sst backup call at {{:337-340}} passes {{true}} for the
> same class of hazard.
> h3. Impact
> A snapshot purge committing between {{Files.list(dbDir)}} and the per file
> read makes the unprotected SST transfer throw {{NoSuchFileException}},
> aborting the whole checkpoint. The {{/dbCheckpoint}} request fails with HTTP
> 500, which affects follower bootstrap and admin tarball download. Externally
> observable, but per request: a later attempt can succeed.
> h3. Reproduction
> {{repro/test_bugMC-3_checkpoint_prelock_sst.sh}} with
> {{repro/java/SpeculaMc3CheckpointPrelockTest.java}}. Evidence in
> {{repro/logs/ev-mc3.log}}.
> h3. Patch
> {{HDDS-XXXXX-MC-3-checkpoint-prelock-sst.patch}} is attached. It applies to
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a
> regression test that was verified to fail without the fix and pass with it.
> {code:none}
> mvn test -pl hadoop-ozone/ozone-manager
> -Dtest=TestOMDBCheckpointServletInodeBasedXferNonLeader
> {code}
> Found by TLA+ model checking and code review of the OM snapshot cache under
> HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated
> with Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
