[
https://issues.apache.org/jira/browse/HDDS-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyesh Karatha reassigned HDDS-16436:
--------------------------------------
Assignee: Priyesh Karatha
> LEGACY bucket lifecycle move to trash with a stale updateID can destructively
> trash a live key
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-16436
> URL: https://issues.apache.org/jira/browse/HDDS-16436
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Priyesh Karatha
> Priority: Critical
> Attachments: HDDS-16436.001.patch
>
>
> h3. Finding
> The LEGACY (non-FSO) rename handler lacks the per-key updateID guard that the
> delete and FSO-rename paths enforce, so a lifecycle move-to-trash carrying a
> scan-captured stale updateID destructively moves the client's live,
> overwritten (or delete+recreated) key into trash where reads return not-found
> and retention eventually purges it; the move is durably committed with no
> automatic recovery. Reachable in the default configuration (move-to-trash
> enabled by default) on a LEGACY bucket with a lifecycle rule and a post-scan
> overwrite. This is silent client-visible data loss.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-4
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-key-lifecycle
> Guidance:
> campaigns/ozone-9fbf9ee/targets/006-om-key-lifecycle/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-key-lifecycle|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # Create a LEGACY bucket with move-to-trash enabled (fs.trash.interval > 0)
> and a lifecycle rule that matches its keys.
> # Write a key K. The lifecycle scan records K and captures its updateID at
> scan time.
> # Before the scan's rename-to-trash commits, overwrite K (a normal concurrent
> write bumps K's updateID), or delete and recreate the bucket so the
> same-named key now has a higher updateID.
> # The lifecycle service issues a RenameKey to move K to .Trash/ with the
> stale scan-captured updateID (KeyLifecycleService.moveToTrash sets
> RenameKeyRequest.updateID to the value captured during the scan).
> # The LEGACY rename handler (OMKeyRenameRequest) resolves the key by name and
> renames it unconditionally, moving the live (overwritten) key into .Trash. A
> client read then returns KEY_NOT_FOUND and trash retention eventually purges
> it.
> The identical stale request is correctly rejected on the delete path
> (OMKeysDeleteRequest re-checks updateID) and on the FSO rename path
> (OMKeyRenameRequestWithFSO re-checks updateID). Only the LEGACY rename path
> is unguarded, so the fencing composition is not inductive.
> h3. Root cause
> Lifecycle stale-write fencing relies on a per-key updateID re-check at commit
> time. That guard is present on the delete path and on the FSO rename handler
> (OMKeyRenameRequestWithFSO.validateAndUpdateCache checks
> renameKeyRequest.hasUpdateID() and rejects with UPDATE_ID_NOT_MATCH when
> fromKeyInfo.getUpdateID() differs), but is absent on the LEGACY (non-FSO)
> OMKeyRenameRequest.validateAndUpdateCache. The LEGACY handler loads fromKey
> by name and renames it without ever reading renameKeyRequest.getUpdateID(),
> so a stale lifecycle rename-to-trash commits against whatever key currently
> holds that name. The FSO guard and the trash-move rename path were added
> together, but the LEGACY variant of the handler was never given the matching
> guard.
> h3. Suggested fix
> Add the same updateID re-check to OMKeyRenameRequest.validateAndUpdateCache:
> after the source key is loaded, when the request carries an updateID, reject
> with UPDATE_ID_NOT_MATCH if the live key's updateID does not match. This
> mirrors the existing FSO guard and makes all three destructive lifecycle
> paths (delete, FSO rename, LEGACY rename) fence stale writes consistently.
> See [^HDDS-16436.001.patch]. The patch adds a targeted stale-updateID
> rejection test and a matching-updateID success test to TestOMKeyRenameRequest
> (LEGACY) and TestOMKeyRenameRequestWithFSO (FSO regression guard).
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
