[
https://issues.apache.org/jira/browse/HDDS-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16436:
------------------------------
Description:
h3. Finding
The LEGACY (non-FSO) rename handler lacks the per-key updateID guard that the
delete and FSO-rename paths enforce, so a lifecycle move-to-trash carrying a
scan-captured stale updateID destructively moves the client's live, overwritten
(or delete+recreated) key into trash where reads return not-found and retention
eventually purges it; the move is durably committed with no automatic recovery.
Reachable in the default configuration (move-to-trash enabled by default) on a
LEGACY bucket with a lifecycle rule and a post-scan overwrite. This is silent
client-visible data loss.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding CR-4
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-key-lifecycle
Guidance:
campaigns/ozone-9fbf9ee/targets/006-om-key-lifecycle/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-key-lifecycle|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
# Create a LEGACY bucket with move-to-trash enabled (fs.trash.interval > 0) and
a lifecycle rule that matches its keys.
# Write a key K. The lifecycle scan records K and captures its updateID at scan
time.
# Before the scan's rename-to-trash commits, overwrite K (a normal concurrent
write bumps K's updateID), or delete and recreate the bucket so the same-named
key now has a higher updateID.
# The lifecycle service issues a RenameKey to move K to .Trash/ with the stale
scan-captured updateID (KeyLifecycleService.moveToTrash sets
RenameKeyRequest.updateID to the value captured during the scan).
# The LEGACY rename handler (OMKeyRenameRequest) resolves the key by name and
renames it unconditionally, moving the live (overwritten) key into .Trash. A
client read then returns KEY_NOT_FOUND and trash retention eventually purges it.
The identical stale request is correctly rejected on the delete path
(OMKeysDeleteRequest re-checks updateID) and on the FSO rename path
(OMKeyRenameRequestWithFSO re-checks updateID). Only the LEGACY rename path is
unguarded, so the fencing composition is not inductive.
h3. Root cause
Lifecycle stale-write fencing relies on a per-key updateID re-check at commit
time. That guard is present on the delete path and on the FSO rename handler
(OMKeyRenameRequestWithFSO.validateAndUpdateCache checks
renameKeyRequest.hasUpdateID() and rejects with UPDATE_ID_NOT_MATCH when
fromKeyInfo.getUpdateID() differs), but is absent on the LEGACY (non-FSO)
OMKeyRenameRequest.validateAndUpdateCache. The LEGACY handler loads fromKey by
name and renames it without ever reading renameKeyRequest.getUpdateID(), so a
stale lifecycle rename-to-trash commits against whatever key currently holds
that name. The FSO guard and the trash-move rename path were added together,
but the LEGACY variant of the handler was never given the matching guard.
h3. Suggested fix
Add the same updateID re-check to OMKeyRenameRequest.validateAndUpdateCache:
after the source key is loaded, when the request carries an updateID, reject
with UPDATE_ID_NOT_MATCH if the live key's updateID does not match. This
mirrors the existing FSO guard and makes all three destructive lifecycle paths
(delete, FSO rename, LEGACY rename) fence stale writes consistently. See
[^HDDS-16436.001.patch]. The patch adds a targeted stale-updateID rejection
test and a matching-updateID success test to TestOMKeyRenameRequest (LEGACY)
and TestOMKeyRenameRequestWithFSO (FSO regression guard).
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
The LEGACY (non-FSO) rename handler lacks the per-key updateID guard that the
delete and FSO-rename paths enforce, so a lifecycle move-to-trash carrying a
scan-captured stale updateID destructively moves the client's live, overwritten
(or delete+recreated) key into trash where reads return not-found and retention
eventually purges it; the move is durably committed with no automatic recovery.
Reachable in the default configuration (move-to-trash enabled by default) on a
LEGACY bucket with a lifecycle rule and a post-scan overwrite. This is silent
client-visible data loss.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding CR-4
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-key-lifecycle
Guidance:
campaigns/ozone-9fbf9ee/targets/006-om-key-lifecycle/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-key-lifecycle|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> LEGACY bucket lifecycle move to trash with a stale updateID can destructively
> trash a live key
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-16436
> URL: https://issues.apache.org/jira/browse/HDDS-16436
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16436.001.patch
>
>
> h3. Finding
> The LEGACY (non-FSO) rename handler lacks the per-key updateID guard that the
> delete and FSO-rename paths enforce, so a lifecycle move-to-trash carrying a
> scan-captured stale updateID destructively moves the client's live,
> overwritten (or delete+recreated) key into trash where reads return not-found
> and retention eventually purges it; the move is durably committed with no
> automatic recovery. Reachable in the default configuration (move-to-trash
> enabled by default) on a LEGACY bucket with a lifecycle rule and a post-scan
> overwrite. This is silent client-visible data loss.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-4
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-key-lifecycle
> Guidance:
> campaigns/ozone-9fbf9ee/targets/006-om-key-lifecycle/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-key-lifecycle|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # Create a LEGACY bucket with move-to-trash enabled (fs.trash.interval > 0)
> and a lifecycle rule that matches its keys.
> # Write a key K. The lifecycle scan records K and captures its updateID at
> scan time.
> # Before the scan's rename-to-trash commits, overwrite K (a normal concurrent
> write bumps K's updateID), or delete and recreate the bucket so the
> same-named key now has a higher updateID.
> # The lifecycle service issues a RenameKey to move K to .Trash/ with the
> stale scan-captured updateID (KeyLifecycleService.moveToTrash sets
> RenameKeyRequest.updateID to the value captured during the scan).
> # The LEGACY rename handler (OMKeyRenameRequest) resolves the key by name and
> renames it unconditionally, moving the live (overwritten) key into .Trash. A
> client read then returns KEY_NOT_FOUND and trash retention eventually purges
> it.
> The identical stale request is correctly rejected on the delete path
> (OMKeysDeleteRequest re-checks updateID) and on the FSO rename path
> (OMKeyRenameRequestWithFSO re-checks updateID). Only the LEGACY rename path
> is unguarded, so the fencing composition is not inductive.
> h3. Root cause
> Lifecycle stale-write fencing relies on a per-key updateID re-check at commit
> time. That guard is present on the delete path and on the FSO rename handler
> (OMKeyRenameRequestWithFSO.validateAndUpdateCache checks
> renameKeyRequest.hasUpdateID() and rejects with UPDATE_ID_NOT_MATCH when
> fromKeyInfo.getUpdateID() differs), but is absent on the LEGACY (non-FSO)
> OMKeyRenameRequest.validateAndUpdateCache. The LEGACY handler loads fromKey
> by name and renames it without ever reading renameKeyRequest.getUpdateID(),
> so a stale lifecycle rename-to-trash commits against whatever key currently
> holds that name. The FSO guard and the trash-move rename path were added
> together, but the LEGACY variant of the handler was never given the matching
> guard.
> h3. Suggested fix
> Add the same updateID re-check to OMKeyRenameRequest.validateAndUpdateCache:
> after the source key is loaded, when the request carries an updateID, reject
> with UPDATE_ID_NOT_MATCH if the live key's updateID does not match. This
> mirrors the existing FSO guard and makes all three destructive lifecycle
> paths (delete, FSO rename, LEGACY rename) fence stale writes consistently.
> See [^HDDS-16436.001.patch]. The patch adds a targeted stale-updateID
> rejection test and a matching-updateID success test to TestOMKeyRenameRequest
> (LEGACY) and TestOMKeyRenameRequestWithFSO (FSO regression guard).
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
