[
https://issues.apache.org/jira/browse/HDDS-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16764:
------------------------------
Attachment: TestBugMC7ExpiredOpenKeyHsyncRace.java
> DeleteOpenKeys removes a key hsync'ed after the cleanup scan and queues its
> blocks for deletion, and the next overwrite commit terminates the OM
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16764
> URL: https://issues.apache.org/jira/browse/HDDS-16764
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments:
> MC-7-keep-open-key-hsynced-after-cleanup-selection.patch,
> TestBugMC7ExpiredOpenKeyHsyncRace.java
>
>
> h3. Mechanism
> {{OmMetadataManagerImpl.getExpiredOpenKeys}} reads hsync state from RocksDB
> only, and {{OMOpenKeysDeleteRequest.updateOpenKeyTableCache}} rechecks only
> existence and {{updateID}}. A first hsync that lands between the cleanup scan
> and the {{DeleteOpenKeys}} request therefore loses its open entry while the
> {{fileTable}} entry keeps {{HSYNC_CLIENT_ID}}.
> {{OMOpenKeysDeleteResponse}} also moves every block of the deleted open key
> to the deleted table, including the blocks the hsync'ed key table entry
> references.
> The commit requests ({{OMKeyCommitRequestWithFSO.java:206}},
> {{OMKeyCommitRequest.java:284}}) then dereference the missing open entry
> ({{openKeyToDelete.toBuilder()}}) without a null check when another client
> commits over that key. {{OMOpenKeysDeleteRequest}} is shared by all bucket
> layouts, so this is not limited to FSO.
> h3. Trigger
> # c1 creates a file and keeps it open past
> {{ozone.om.open.key.expire.threshold}} without hsync.
> # c2 opens the same path (overwrite of an hsync'ed file is allowed).
> # The cleanup scan selects c1's open key.
> # c1's first hsync is applied before {{DeleteOpenKeys}}.
> # c2 commits.
> h3. Impact
> c1's hsync is acknowledged but its open entry is deleted and the blocks of
> the hsync'ed data are queued for deletion, with no second writer needed
> (verified up to the deleted table entry; the physical block deletion was not
> run). c2's commit then throws {{NullPointerException}} in apply and
> {{OzoneManagerStateMachine.runCommand}} terminates the OM. The entry is
> already in the Raft log, so the other OMs and a restart would hit it again
> (argued from the deterministic apply path, not run).
> h3. Reproduction
> PASS, 3 of 3 runs. [^TestBugMC7ExpiredOpenKeyHsyncRace.java]
> ({{hadoop-ozone/ozone-manager}}) uses only client calls against one real OM
> with its real {{OpenKeyCleanupService}}, with the service interval and expiry
> threshold shortened by configuration. With defaults the window is
> milliseconds, once per service run, on a file open for seven days.
> h3. Suggested fix
> * Close the race in {{OMOpenKeysDeleteRequest.updateOpenKeyTableCache}}:
> never delete an open key that a committed key still points to through
> {{HSYNC_CLIENT_ID}}.
> * Null check {{openKeyToDelete}} in both commit requests. This removes the
> termination and lets a cluster already in this state recover.
> HDDS-16732 reports the same cleanup site enqueueing a live key's blocks on a
> versioned bucket; different trigger, same missing key table check.
> h3. Patch
> [^MC-7-keep-open-key-hsynced-after-cleanup-selection.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> 9c6bee807c8.
> Two changes:
> * {{OMOpenKeysDeleteRequest}}: under the bucket lock, an open key whose own
> client ID is in its {{HSYNC_CLIENT_ID}} is looked up in the key table and
> skipped when the committed key still carries that client ID. A deleted key
> has no entry and an overwritten key carries another writer's ID or none, so
> both are still deleted. The extra read only happens for open keys that are
> hsync'ed, which the scan normally never submits.
> * {{OMKeyCommitRequest}} and {{OMKeyCommitRequestWithFSO}}: the
> {{OVERWRITTEN_HSYNC_KEY}} marking is skipped when the open entry is missing,
> so an OM that already holds such a key no longer terminates on the next
> overwrite commit.
> Covered by {{testDeleteHsyncedKeys}} in the existing
> {{TestOMOpenKeysDeleteRequest}} (both layouts) and
> {{testOverwriteHsyncKeyWithoutOpenKey}} in {{TestOMKeyCommitRequest}}
> (inherited by the FSO variant). Without the change the first fails because
> the hsync'ed open key is gone, and the second with the
> {{NullPointerException}} on {{openKeyToDelete}}. The open key delete, key
> commit and cleanup service unit suites (82 tests) and {{TestHSync}} (42
> tests) pass, checkstyle is clean. The cross OM and restart behaviour is
> argued from the apply path, not run.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
