[
https://issues.apache.org/jira/browse/HDDS-16847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16847:
------------------------------
Attachment: TestBugCR1AutoCommitModificationTime.java
> Hsync key committed by OpenKeyCleanupService after the lease hard limit is
> stored with modification time 0
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16847
> URL: https://issues.apache.org/jira/browse/HDDS-16847
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-1-set-modification-time-on-hsync-auto-commit.patch,
> TestBugCR1AutoCommitModificationTime.java
>
>
> h3. Mechanism
> When an hsync'ed open key passes {{ozone.om.lease.hard.limit}},
> {{OpenKeyCleanupService}} commits it.
> {{OmMetadataManagerImpl.getExpiredOpenKeys}} builds the {{KeyArgs}} of that
> {{CommitKey}} request with volume, bucket, key name, size, block locations
> and replication, and {{OpenKeyCleanupService.createCommitKeyRequest}} submits
> it with {{OzoneManagerRatisUtils.submitRequest}}, which does not run
> {{preExecute}}. {{OMKeyCommitRequest.preExecute}} is where a commit gets
> {{KeyArgs.modificationTime = Time.now()}}, so the field stays at its default
> 0. {{OMKeyCommitRequest.validateAndUpdateCache}} and
> {{OMKeyCommitRequestWithFSO.validateAndUpdateCache}} copy it into the
> committed key, which is then written to the key or file table with
> modification time 0 (1970-01-01).
> This exists since the auto-commit was added in HDDS-7782 (ozone-1.4.0);
> HDDS-10141 moved it to the hard limit without changing the request.
> {{TestOpenKeyCleanupService.testCommitExpiredHsyncKeys}} checks only the key
> name after the auto-commit.
> The other things {{preExecute}} does for a commit (bucket link resolution,
> key name normalization, write ACL check on the open key) are not needed here:
> volume, bucket and key name come from rows that the create request already
> resolved and normalized, and the ACL check is skipped for this internal
> request in the same way as for the {{DeleteOpenKeys}} request of the same
> service.
> Related: HDDS-16825 (same auto-commit path, different defect: when the key is
> committed and with which length, not the stored time).
> h3. Trigger
> # {{ozone.fs.hsync.enabled=true}}. A client opens a key, writes and calls
> hsync.
> # The client stops without closing the file (for example it crashes) and
> nobody recovers the lease.
> # After {{ozone.om.lease.hard.limit}} (default 7d) {{OpenKeyCleanupService}}
> commits the key.
> h3. Impact
> * Every key committed this way has modification time 0 and keeps it until it
> is overwritten. Shown with {{OzoneManagerProtocol.getKeyInfo}} for FSO and
> OBS buckets. File system status and S3 listings read the same field (code
> reading, not run). The patch does not repair keys that were committed this
> way before it; they can be found by their modification time 0, which key info
> and key listings show as 1970-01-01T00:00:00Z (code reading, not run).
> * A lifecycle expiration rule based on days matches the key at once: a
> validated "expire after 30 days" rule returns true from the expiration check
> in {{OmLCRule.match}} that {{KeyLifecycleService}} uses before deleting or
> moving a key to trash, while a key committed normally in the same bucket does
> not match. With a date based rule the difference only shows for keys
> auto-committed after that date (code reading, not run).
> {{KeyLifecycleService}} itself was not run; it is not in a release yet (added
> by HDDS-12780 after ozone-2.2.1) and is off by default
> ({{ozone.lifecycle.service.enabled=false}}). In released versions the effect
> is the wrong modification time alone.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugCR1AutoCommitModificationTime.java] ({{hadoop-ozone/ozone-manager}})
> runs one real OM through {{OmTestManagers}}, with the hard limit shortened to
> 200 ms, for an FSO and an OBS bucket. It opens a key, allocates a block and
> calls hsync through {{OzoneManagerProtocol}}, commits a control key normally,
> and lets the real {{OpenKeyCleanupService}} commit the abandoned key. A
> passing test means the defect is present. Output for FSO (OBS is the same):
> {noformat}
> FILE_SYSTEM_OPTIMIZED: testStart=1791649284083 hsync mtime=1791649284436
> autoCommitStart=1791649284760 auto-committed mtime=0 control
> mtime=1791649284500 30-day rule matches: auto-committed=true control=false
> {noformat}
> h3. Patch
> [^CR-1-set-modification-time-on-hsync-auto-commit.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{OpenKeyCleanupService.createCommitKeyRequest}} sets
> {{KeyArgs.modificationTime}} to {{Time.now()}} before submitting, as
> {{preExecute}} does on the client path. The value is chosen before the
> request enters Ratis, so all OMs apply the same value, and the request format
> and apply logic are unchanged. Running the full {{preExecute}} (as
> {{KeyLifecycleService}} does for its requests) was not chosen, because it
> would add a write ACL check under the OM's own user and other client checks
> to this internal request; a failing check would leave the hsync key open.
> {{TestOpenKeyCleanupService.testCommitExpiredHsyncKeys}} now asserts that the
> auto-committed keys have a modification time at or after the moment the
> service was resumed. Without the change it fails with all 10 keys at 0; with
> it {{TestOpenKeyCleanupService}}, {{TestOMKeyCommitRequest}},
> {{TestOMKeyCommitRequestWithFSO}} and {{TestOmMetadataManager}} pass (86
> tests), as does {{TestHSync}} on a mini cluster (42 tests), and checkstyle is
> clean. With the patch the attached reproduction fails, because the
> auto-committed key gets the commit time and no longer matches the 30 day rule.
> Found by code review of the OM open key cleanup and hsync lease recovery
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
