Siyao Meng created HDDS-16849:
---------------------------------
Summary: OBS/LEGACY key rename does not reject an hsync'ed key as
FSO does
Key: HDDS-16849
URL: https://issues.apache.org/jira/browse/HDDS-16849
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-reject-rename-of-hsynced-key-in-obs-and-legacy-buckets.patch
h3. Mechanism
A key that a writer has hsync'ed has two rows: its row in the key table, which
carries the hsync state ({{OmKeyInfo.isHsync}}), and the writer's open key row,
both under the key's name. On FILE_SYSTEM_OPTIMIZED buckets
{{OMKeyRenameRequestWithFSO}} rejects renaming such a key with
{{RENAME_OPEN_FILE}} (HDDS-8545). On OBJECT_STORE and LEGACY buckets
{{OMKeyRenameRequest}} and {{OMKeysRenameRequest}} have no such check, only
{{// TODO: Add support for renaming open key}}. They move the key table row,
hsync state included, to the new name and leave the writer's open key row under
the old name.
{{OmMetadataManagerImpl.getExpiredOpenKeys}} relies on the two rows matching.
For an open key row past {{ozone.om.lease.hard.limit}} that is hsync'ed, it
reads the key table row under the same name, and after such a rename there is
none, so the call throws. {{OpenKeyCleanupService.OpenKeyCleanupTask.call}}
only catches {{IOException}}, so the DEFAULT layout run fails before it submits
anything.
h3. Trigger
Hsync enabled on the OM ({{ozone.hbase.enhancements.allowed=true}} and
{{ozone.fs.hsync.enabled=true}}, both false by default) and an OBJECT_STORE or
LEGACY bucket:
# A writer opens a key, writes and calls hsync.
# The key is renamed while the writer still has it open
({{OzoneBucket.renameKey}} or {{renameKeys}}, {{ozone sh key rename}}, or an
ofs/o3fs rename).
# The writer stops (dies, or stays idle without another hsync or close) until
its open key is past the lease hard limit (default 7d).
h3. Impact
* Open key cleanup for OBS and LEGACY buckets fails on that OM. Each DEFAULT
layout run that reaches the leftover open key row fails, once the expired keys
sorted in front of it are fewer than
{{ozone.om.open.key.cleanup.limit.per.task}} (default 1000). Expired open keys
sorted after the row are never cleaned, and expired hsync'ed keys are not auto
committed. FSO buckets are not affected.
* It lasts until the writer resumes: an hsync or close under the old name
brings the two rows back in line. A writer that has died never does, and lease
recovery only handles FSO.
h3. Reproduction
There is no standalone reproduction. The regression tests in the attached patch
set the hsync state of the source key internally, as the existing FSO test
does, and fail on unmodified source:
* {{TestOMKeyRenameRequest.testRenameOpenFile}}
({{hadoop-ozone/ozone-manager}}): renaming an hsync'ed key must fail with
{{RENAME_OPEN_FILE}} and leave the source in place. Without the change it fails
with {{expected: <RENAME_OPEN_FILE> but was: <OK>}}.
* {{TestOMKeysRenameRequest.testKeysRenameRequestWithOpenFile}}: in a
{{RenameKeys}} request with ten plain keys and one hsync'ed key, the hsync'ed
key must be the only pair not renamed. Without the change it fails with
{{expected: <PARTIAL_RENAME> but was: <OK>}}.
h3. Patch
[^MC-3-reject-rename-of-hsynced-key-in-obs-and-legacy-buckets.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
{{OMKeyRenameRequest}} now fails with {{RENAME_OPEN_FILE}} when the source key
is hsync'ed, with the message used by {{OMKeyRenameRequestWithFSO}}.
{{OMKeysRenameRequest}} reports such a pair in {{unRenamedKeys}} and returns
{{PARTIAL_RENAME}}, as it does for other pairs it cannot rename. Keys that are
not hsync'ed, closed keys and FSO buckets are unaffected.
The check changes what the apply writes, so it is enabled by a new
{{OzoneManagerVersion.RENAME_REJECTS_HSYNC_KEY}} (101) through
{{OMVersionManager.isAllowed}}. Until the cluster is finalized to that version
every OM applies the rename as before, so OMs on different software versions
never apply the same log entry differently, and the defect remains until
finalization. The version number is a placeholder to be agreed at merge, since
other open patches also add an OM version 101. {{testRenameOpenFile}} and
{{TestOMKeysRenameRequest.testKeysRenameRequestWithOpenFileBeforeFinalization}}
check that an OM that is not finalized still applies the rename.
Visible change with hsync enabled, after finalization: renaming a key that is
still being written with hsync fails instead of succeeding. An o3fs/ofs
directory rename on a LEGACY bucket renames key by key and stops at such a
file, so the files renamed before it stay at the destination. HBase on Ozone
needs FSO buckets, because lease recovery is FSO only, and there this rename is
already rejected (HDDS-8545).
The patch does not repair keys renamed before it. For such a leftover open key
row, the renamed key still references the row's blocks, so any change that
handles the row in the cleanup scan should skip it rather than delete it.
With the patch the key rename request and response suites,
{{TestOpenKeyCleanupService}}, {{TestOzoneManagerRequestHandler}},
{{TestBucketLayoutAwareOMKeyFactory}}, the OM version and finalization unit
suites and the OM upgrade integration tests, including
{{TestOMUpgradeFinalization}}, pass, and checkstyle on {{hadoop-hdds/common}}
and {{hadoop-ozone/ozone-manager}} is clean.
Found by TLA+ model checking and code review of the OM open key cleanup and
hsync lease recovery paths, as part of the TLA+ verification effort under
HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against
HDDS issues and apache/ozone pull requests for duplicates before filing.
Related but not duplicates: HDDS-8545 (added the rejection for FSO), HDDS-10631
(proposal to block every open key rename, no OBS/LEGACY or cleanup symptom).
The attached patch is a proposal for review. Generated with Specula (Claude
Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
