Siyao Meng created HDDS-16847:
---------------------------------
Summary: Hsync key committed by OpenKeyCleanupService after the
lease hard limit is stored with modification time 0
Key: HDDS-16847
URL: https://issues.apache.org/jira/browse/HDDS-16847
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-set-modification-time-on-hsync-auto-commit.patch,
TestBugCR1AutoCommitModificationTime.java
h3. Mechanism
When an hsync'ed open key passes {{ozone.om.lease.hard.limit}},
{{OpenKeyCleanupService}} commits it.
{{OmMetadataManagerImpl.getExpiredOpenKeys}} builds the {{KeyArgs}} of that
{{CommitKey}} request with volume, bucket, key name, size, block locations and
replication, and {{OpenKeyCleanupService.createCommitKeyRequest}} submits it
with {{OzoneManagerRatisUtils.submitRequest}}, which does not run
{{preExecute}}. {{OMKeyCommitRequest.preExecute}} is where a commit gets
{{KeyArgs.modificationTime = Time.now()}}, so the field stays at its default 0.
{{OMKeyCommitRequest.validateAndUpdateCache}} and
{{OMKeyCommitRequestWithFSO.validateAndUpdateCache}} copy it into the committed
key, which is then written to the key or file table with modification time 0
(1970-01-01).
This exists since the auto-commit was added in HDDS-7782 (ozone-1.4.0);
HDDS-10141 moved it to the hard limit without changing the request.
{{TestOpenKeyCleanupService.testCommitExpiredHsyncKeys}} checks only the key
name after the auto-commit.
The other things {{preExecute}} does for a commit (bucket link resolution, key
name normalization, write ACL check on the open key) are not needed here:
volume, bucket and key name come from rows that the create request already
resolved and normalized, and the ACL check is skipped for this internal request
in the same way as for the {{DeleteOpenKeys}} request of the same service.
Related: HDDS-16825 (same auto-commit path, different defect: when the key is
committed and with which length, not the stored time).
h3. Trigger
# {{ozone.fs.hsync.enabled=true}}. A client opens a key, writes and calls hsync.
# The client stops without closing the file (for example it crashes) and nobody
recovers the lease.
# After {{ozone.om.lease.hard.limit}} (default 7d) {{OpenKeyCleanupService}}
commits the key.
h3. Impact
* Every key committed this way has modification time 0 and keeps it until it is
overwritten. Shown with {{OzoneManagerProtocol.getKeyInfo}} for FSO and OBS
buckets. File system status and S3 listings read the same field (code reading,
not run). The patch does not repair keys that were committed this way before
it; they can be found by their modification time 0, which key info and key
listings show as 1970-01-01T00:00:00Z (code reading, not run).
* A lifecycle expiration rule based on days matches the key at once: a
validated "expire after 30 days" rule returns true from the expiration check in
{{OmLCRule.match}} that {{KeyLifecycleService}} uses before deleting or moving
a key to trash, while a key committed normally in the same bucket does not
match. With a date based rule the difference only shows for keys auto-committed
after that date (code reading, not run). {{KeyLifecycleService}} itself was not
run; it is not in a release yet (added by HDDS-12780 after ozone-2.2.1) and is
off by default ({{ozone.lifecycle.service.enabled=false}}). In released
versions the effect is the wrong modification time alone.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR1AutoCommitModificationTime.java] ({{hadoop-ozone/ozone-manager}})
runs one real OM through {{OmTestManagers}}, with the hard limit shortened to
200 ms, for an FSO and an OBS bucket. It opens a key, allocates a block and
calls hsync through {{OzoneManagerProtocol}}, commits a control key normally,
and lets the real {{OpenKeyCleanupService}} commit the abandoned key. A passing
test means the defect is present. Output for FSO (OBS is the same):
{noformat}
FILE_SYSTEM_OPTIMIZED: testStart=1791649284083 hsync mtime=1791649284436
autoCommitStart=1791649284760 auto-committed mtime=0 control
mtime=1791649284500 30-day rule matches: auto-committed=true control=false
{noformat}
h3. Patch
[^CR-1-set-modification-time-on-hsync-auto-commit.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
{{OpenKeyCleanupService.createCommitKeyRequest}} sets
{{KeyArgs.modificationTime}} to {{Time.now()}} before submitting, as
{{preExecute}} does on the client path. The value is chosen before the request
enters Ratis, so all OMs apply the same value, and the request format and apply
logic are unchanged. Running the full {{preExecute}} (as
{{KeyLifecycleService}} does for its requests) was not chosen, because it would
add a write ACL check under the OM's own user and other client checks to this
internal request; a failing check would leave the hsync key open.
{{TestOpenKeyCleanupService.testCommitExpiredHsyncKeys}} now asserts that the
auto-committed keys have a modification time at or after the moment the service
was resumed. Without the change it fails with all 10 keys at 0; with it
{{TestOpenKeyCleanupService}}, {{TestOMKeyCommitRequest}},
{{TestOMKeyCommitRequestWithFSO}} and {{TestOmMetadataManager}} pass (86
tests), as does {{TestHSync}} on a mini cluster (42 tests), and checkstyle is
clean. With the patch the attached reproduction fails, because the
auto-committed key gets the commit time and no longer matches the 30 day rule.
Found by code review of the OM open key cleanup and hsync lease recovery paths,
as part of the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
