Siyao Meng created HDDS-16764:
---------------------------------
Summary: DeleteOpenKeys removes a key hsync'ed after the cleanup
scan and queues its blocks for deletion, and the next overwrite commit
terminates the OM
Key: HDDS-16764
URL: https://issues.apache.org/jira/browse/HDDS-16764
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-7-keep-open-key-hsynced-after-cleanup-selection.patch,
TestBugMC7ExpiredOpenKeyHsyncRace.java
h3. Mechanism
{{OmMetadataManagerImpl.getExpiredOpenKeys}} reads hsync state from RocksDB
only, and {{OMOpenKeysDeleteRequest.updateOpenKeyTableCache}} rechecks only
existence and {{updateID}}. A first hsync that lands between the cleanup scan
and the {{DeleteOpenKeys}} request therefore loses its open entry while the
{{fileTable}} entry keeps {{HSYNC_CLIENT_ID}}.
{{OMOpenKeysDeleteResponse}} also moves every block of the deleted open key to
the deleted table, including the blocks the hsync'ed key table entry references.
The commit requests ({{OMKeyCommitRequestWithFSO.java:206}},
{{OMKeyCommitRequest.java:284}}) then dereference the missing open entry
({{openKeyToDelete.toBuilder()}}) without a null check when another client
commits over that key. {{OMOpenKeysDeleteRequest}} is shared by all bucket
layouts, so this is not limited to FSO.
h3. Trigger
# c1 creates a file and keeps it open past
{{ozone.om.open.key.expire.threshold}} without hsync.
# c2 opens the same path (overwrite of an hsync'ed file is allowed).
# The cleanup scan selects c1's open key.
# c1's first hsync is applied before {{DeleteOpenKeys}}.
# c2 commits.
h3. Impact
c1's hsync is acknowledged but its open entry is deleted and the blocks of the
hsync'ed data are queued for deletion, with no second writer needed (verified
up to the deleted table entry; the physical block deletion was not run). c2's
commit then throws {{NullPointerException}} in apply and
{{OzoneManagerStateMachine.runCommand}} terminates the OM. The entry is already
in the Raft log, so the other OMs and a restart would hit it again (argued from
the deterministic apply path, not run).
h3. Reproduction
PASS, 3 of 3 runs. [^TestBugMC7ExpiredOpenKeyHsyncRace.java]
({{hadoop-ozone/ozone-manager}}) uses only client calls against one real OM
with its real {{OpenKeyCleanupService}}, with the service interval and expiry
threshold shortened by configuration. With defaults the window is milliseconds,
once per service run, on a file open for seven days.
h3. Suggested fix
* Close the race in {{OMOpenKeysDeleteRequest.updateOpenKeyTableCache}}: never
delete an open key that a committed key still points to through
{{HSYNC_CLIENT_ID}}.
* Null check {{openKeyToDelete}} in both commit requests. This removes the
termination and lets a cluster already in this state recover.
HDDS-16732 reports the same cleanup site enqueueing a live key's blocks on a
versioned bucket; different trigger, same missing key table check.
h3. Patch
[^MC-7-keep-open-key-hsynced-after-cleanup-selection.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
9c6bee807c8.
Two changes:
* {{OMOpenKeysDeleteRequest}}: under the bucket lock, an open key whose own
client ID is in its {{HSYNC_CLIENT_ID}} is looked up in the key table and
skipped when the committed key still carries that client ID. A deleted key has
no entry and an overwritten key carries another writer's ID or none, so both
are still deleted. The extra read only happens for open keys that are hsync'ed,
which the scan normally never submits.
* {{OMKeyCommitRequest}} and {{OMKeyCommitRequestWithFSO}}: the
{{OVERWRITTEN_HSYNC_KEY}} marking is skipped when the open entry is missing, so
an OM that already holds such a key no longer terminates on the next overwrite
commit.
Covered by {{testDeleteHsyncedKeys}} in the existing
{{TestOMOpenKeysDeleteRequest}} (both layouts) and
{{testOverwriteHsyncKeyWithoutOpenKey}} in {{TestOMKeyCommitRequest}}
(inherited by the FSO variant). Without the change the first fails because the
hsync'ed open key is gone, and the second with the {{NullPointerException}} on
{{openKeyToDelete}}. The open key delete, key commit and cleanup service unit
suites (82 tests) and {{TestHSync}} (42 tests) pass, checkstyle is clean. The
cross OM and restart behaviour is argued from the apply path, not run.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
