[
https://issues.apache.org/jira/browse/HDDS-16775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16775:
------------------------------
Attachment:
CR-4-commit-expired-hsync-key-at-current-path-after-parent-rename.patch
> Renaming an ancestor of an hsync'ed file breaks its lease expiry auto commit
> ----------------------------------------------------------------------------
>
> Key: HDDS-16775
> URL: https://issues.apache.org/jira/browse/HDDS-16775
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-4-commit-expired-hsync-key-at-current-path-after-parent-rename.patch
>
>
> h3. Mechanism
> {{OMKeyRenameRequestWithFSO}} rejects renaming an hsync'ed file (HDDS-8545)
> but not a directory above one, and does not touch the open file table. The
> open row keeps the create time full path in {{keyName}}.
> {{OmMetadataManagerImpl.getExpiredOpenKeys}} copies that path into the lease
> expiry auto commit request, and {{OMKeyCommitRequestWithFSO}} resolves it by
> path.
> h3. Trigger
> With hsync enabled on an FSO bucket: create and hsync {{/v/b/src/sub/f}},
> rename {{/v/b/src}} to {{/v/b/dst}}, and the writer does not close before
> {{ozone.om.lease.hard.limit}}.
> h3. Impact
> Every {{OpenKeyCleanupService}} run submits a {{CommitKey}} for {{src/sub/f}}
> that fails with {{DIRECTORY_NOT_FOUND}}, and the service takes no action on
> the failed response (the failure is logged at ERROR by the commit request on
> every run). The entry is returned by every scan, so it also uses up part of
> each run's {{ozone.om.open.key.cleanup.limit.per.task}} budget. The file
> keeps its hsync client ID and its open row, so it stays an open file after
> lease expiry and renaming it is rejected with {{RENAME_OPEN_FILE}}. No data
> is lost. A manual {{recoverLease}} on the new path clears it.
> h3. Reproduction
> PASS, with a test added to the existing hsync mini cluster suite
> ({{TestHSync}}): the three steps above, then run the cleanup service and
> assert the file is still open. It also reproduces at OM level through the
> real write path without a mini cluster.
> h3. Suggested fix
> Make the auto commit independent of the stored path: rebuild the current path
> from the parent ID in {{getExpiredOpenKeys}}, or locate the file by parent
> object ID in the commit. Rejecting the rename of a directory that has
> hsync'ed open descendants is not a good alternative: it needs a subtree check
> on every directory rename, has no precedent (HDDS-10650 lets a recursive
> delete through and flags the descendants' open keys), and HBase WAL splitting
> renames a directory whose WALs are still open.
> h3. Patch
> [^CR-4-commit-expired-hsync-key-at-current-path-after-parent-rename.patch],
> against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master
> at 9c6bee807c8.
> In {{getExpiredOpenKeys}}, for an FSO hsync'ed key that is about to be auto
> committed, the stored path is first checked with
> {{OMFileRequest.getParentID}}; if it still leads to the open key's parent ID
> it is used as before. Only when it does not, the parent's current path is
> rebuilt with the existing {{FSODirectoryPathResolver}} and the commit request
> is built with that path. The commit request, the proto and the apply path are
> unchanged. Directory renames stay allowed, and the file is now closed at its
> current path once the lease hard limit passes.
> Cost: one directory table lookup per path component for each expired hsync'ed
> key, and for a key whose stored path is stale a breadth first walk of the
> bucket's directory tree per run. Not addressed: the writer's own hsync or
> close after a parent rename still fails with {{DIRECTORY_NOT_FOUND}}, because
> the client sends the path it opened.
> Covered by {{testCommitExpiredHsyncKeyAfterParentRename}} in the existing
> {{TestOpenKeyCleanupService}}, which fails without the change because the
> scan asks to commit {{src/sub/file}} where {{dst/sub/file}} is expected.
> {{TestOpenKeyCleanupService}}, {{TestOmMetadataManager}} and
> {{TestFSODirectoryPathResolver}} (46 tests) and {{TestHSync}} (42 tests)
> pass, checkstyle is clean.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
