Siyao Meng created HDDS-16768:
---------------------------------
Summary: Single key delete of an hsync'ed FSO file persists
DELETED_HSYNC_KEY under the wrong open table key
Key: HDDS-16768
URL: https://issues.apache.org/jira/browse/HDDS-16768
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-1-persist-deleted-hsync-marker-under-fso-open-key.patch
h3. Mechanism
{{OMKeyDeleteRequestWithFSO.validateAndUpdateCache}} flags the hsync writer's
open entry with {{DELETED_HSYNC_KEY}} in the cache under the FSO key,
{{getOpenFileName(volumeId, bucketId, parentId, fileName, hsyncClientId)}}
({{:174}}). {{OMKeyDeleteResponseWithFSO.addToDBBatch}} persists that flagged
entry under the non FSO key, {{getOpenKey(volume, bucket, key, hsyncClientId)}}
({{:117-118}}).
The real FSO open row in RocksDB therefore stays unflagged, and a stray
{{/volume/bucket/key/clientId}} row is added to the open file table.
h3. Trigger
On an FSO bucket with hsync enabled: create a file, hsync, delete the file non
recursively ({{FileSystem.delete(file, false)}}), and let the delete flush and
its cache epoch evict (automatic, milliseconds).
h3. Impact
Two consequences of the same root cause, both reproduced:
* *Deleted file comes back.* When the writer then closes or hsyncs, its
{{CommitKey}} is accepted instead of failing with {{KEY_NOT_FOUND}}. The
deleted file is back in the file table and survives an OM restart, while its
earlier blocks are already queued for deletion by the delete. This is a
regression of HDDS-9930 for FSO single key delete, introduced when HDDS-10650
replaced the removal of the open key with the {{DELETED_HSYNC_KEY}} marker.
* *Open key cleanup stops.* If the writer never closes,
{{OmMetadataManagerImpl.getExpiredOpenKeys}} reaches the unflagged hsync'ed
open row after the lease hard limit, looks up its {{fileTable}} row, gets null,
and dereferences it. {{OpenKeyCleanupService}} catches only {{IOException}}, so
the FSO task of every run that reaches the row dies with
{{NullPointerException}} ({{OmMetadataManagerImpl.java:1516}}) and submits
nothing. The row is never removed, so a restart does not help, and expired FSO
open keys are no longer cleaned up or auto committed.
h3. Reproduction
PASS, no injected timing, through the public FileSystem API on a single OM mini
cluster for the first consequence, and with FileSystem calls plus the real
cleanup service for the second. Both also reproduce at OM level through the
real write path without a mini cluster. Divergence between OMs is argued from
the same mechanism, not run.
h3. Suggested fix
* In {{OMKeyDeleteResponseWithFSO.addToDBBatch}}, persist
{{deletedOpenKeyInfo}} under the key the request used for the cache
({{getOpenFileName(...)}}), or pass the request's {{dbOpenKey}} into the
response.
* Harden {{getExpiredOpenKeys}}: when the {{fileTable}} lookup returns null for
an hsync'ed open row, treat the row as an orphan and add it to the delete list.
This also heals rows already persisted.
* The sibling sites {{OmKeysDeleteRequestWithFSO}} and
{{OMDirectoriesPurgeRequestWithFSO}} with their responses were checked and
persist under the right key.
* Existing clusters may hold stray {{/volume/bucket/key/clientId}} rows in the
open file table. Those carry the marker, so the cleanup service removes them
once they expire, provided the scan is not stopped first by an unflagged row.
h3. Patch
[^MC-1-persist-deleted-hsync-marker-under-fso-open-key.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
9c6bee807c8.
Two changes:
* Root cause: {{OMKeyDeleteResponseWithFSO.addToDBBatch}} builds the open key
with {{getOpenFileName(volumeId, bucketId, parentObjectID, fileName,
hsyncClientId)}}, the key the request used for the cache.
* Scan hardening: in {{getExpiredOpenKeys}}, an expired hsync'ed open key whose
key table entry is missing is added to the delete list instead of being
dereferenced. This is what lets a cluster that already holds unflagged rows
recover. It is separable from the root cause change if a reviewer prefers two
patches.
Covered by {{testCleanupDeletedHsyncKey}} and
{{testCleanupHsyncKeyWithoutCommittedFile}} in the existing
{{TestOpenKeyCleanupService}}. Without the change the first fails because the
writer's commit after the delete is accepted ("Expected OMException to be
thrown, but nothing was thrown"), and the scan throws the
{{NullPointerException}} above. The cleanup service, key delete, open key
delete, directory purge and {{TestOmMetadataManager}} unit suites (143 tests)
and {{TestHSync}} (42 tests) pass, checkstyle is clean.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
