[
https://issues.apache.org/jira/browse/HDDS-16768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16768:
------------------------------
Attachment: MC-1-persist-deleted-hsync-marker-under-fso-open-key.patch
> Single key delete of an hsync'ed FSO file persists DELETED_HSYNC_KEY under
> the wrong open table key
> ---------------------------------------------------------------------------------------------------
>
> Key: HDDS-16768
> URL: https://issues.apache.org/jira/browse/HDDS-16768
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments:
> MC-1-persist-deleted-hsync-marker-under-fso-open-key.patch
>
>
> h3. Mechanism
> {{OMKeyDeleteRequestWithFSO.validateAndUpdateCache}} flags the hsync writer's
> open entry with {{DELETED_HSYNC_KEY}} in the cache under the FSO key,
> {{getOpenFileName(volumeId, bucketId, parentId, fileName, hsyncClientId)}}
> ({{:174}}). {{OMKeyDeleteResponseWithFSO.addToDBBatch}} persists that flagged
> entry under the non FSO key, {{getOpenKey(volume, bucket, key,
> hsyncClientId)}} ({{:117-118}}).
> The real FSO open row in RocksDB therefore stays unflagged, and a stray
> {{/volume/bucket/key/clientId}} row is added to the open file table.
> h3. Trigger
> On an FSO bucket with hsync enabled: create a file, hsync, delete the file
> non recursively ({{FileSystem.delete(file, false)}}), and let the delete
> flush and its cache epoch evict (automatic, milliseconds).
> h3. Impact
> Two consequences of the same root cause, both reproduced:
> * *Deleted file comes back.* When the writer then closes or hsyncs, its
> {{CommitKey}} is accepted instead of failing with {{KEY_NOT_FOUND}}. The
> deleted file is back in the file table and survives an OM restart, while its
> earlier blocks are already queued for deletion by the delete. This is a
> regression of HDDS-9930 for FSO single key delete, introduced when HDDS-10650
> replaced the removal of the open key with the {{DELETED_HSYNC_KEY}} marker.
> * *Open key cleanup stops.* If the writer never closes,
> {{OmMetadataManagerImpl.getExpiredOpenKeys}} reaches the unflagged hsync'ed
> open row after the lease hard limit, looks up its {{fileTable}} row, gets
> null, and dereferences it. {{OpenKeyCleanupService}} catches only
> {{IOException}}, so the FSO task of every run that reaches the row dies with
> {{NullPointerException}} ({{OmMetadataManagerImpl.java:1516}}) and submits
> nothing. The row is never removed, so a restart does not help, and expired
> FSO open keys are no longer cleaned up or auto committed.
> h3. Reproduction
> PASS, no injected timing, through the public FileSystem API on a single OM
> mini cluster for the first consequence, and with FileSystem calls plus the
> real cleanup service for the second. Both also reproduce at OM level through
> the real write path without a mini cluster. Divergence between OMs is argued
> from the same mechanism, not run.
> h3. Suggested fix
> * In {{OMKeyDeleteResponseWithFSO.addToDBBatch}}, persist
> {{deletedOpenKeyInfo}} under the key the request used for the cache
> ({{getOpenFileName(...)}}), or pass the request's {{dbOpenKey}} into the
> response.
> * Harden {{getExpiredOpenKeys}}: when the {{fileTable}} lookup returns null
> for an hsync'ed open row, treat the row as an orphan and add it to the delete
> list. This also heals rows already persisted.
> * The sibling sites {{OmKeysDeleteRequestWithFSO}} and
> {{OMDirectoriesPurgeRequestWithFSO}} with their responses were checked and
> persist under the right key.
> * Existing clusters may hold stray {{/volume/bucket/key/clientId}} rows in
> the open file table. Those carry the marker, so the cleanup service removes
> them once they expire, provided the scan is not stopped first by an unflagged
> row.
> h3. Patch
> [^MC-1-persist-deleted-hsync-marker-under-fso-open-key.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> 9c6bee807c8.
> Two changes:
> * Root cause: {{OMKeyDeleteResponseWithFSO.addToDBBatch}} builds the open key
> with {{getOpenFileName(volumeId, bucketId, parentObjectID, fileName,
> hsyncClientId)}}, the key the request used for the cache.
> * Scan hardening: in {{getExpiredOpenKeys}}, an expired hsync'ed open key
> whose key table entry is missing is added to the delete list instead of being
> dereferenced. This is what lets a cluster that already holds unflagged rows
> recover. It is separable from the root cause change if a reviewer prefers two
> patches.
> Covered by {{testCleanupDeletedHsyncKey}} and
> {{testCleanupHsyncKeyWithoutCommittedFile}} in the existing
> {{TestOpenKeyCleanupService}}. Without the change the first fails because the
> writer's commit after the delete is accepted ("Expected OMException to be
> thrown, but nothing was thrown"), and the scan throws the
> {{NullPointerException}} above. The cleanup service, key delete, open key
> delete, directory purge and {{TestOmMetadataManager}} unit suites (143 tests)
> and {{TestHSync}} (42 tests) pass, checkstyle is clean.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
