[
https://issues.apache.org/jira/browse/HDDS-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17251:
------------------------------
Attachment: CR-15-list-open-multipart-uploads-in-list-open-files.patch
> ozone admin om list-open-files fails with NumberFormatException when an open
> multipart upload is in the listed range
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17251
> URL: https://issues.apache.org/jira/browse/HDDS-17251
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-15-list-open-multipart-uploads-in-list-open-files.patch,
> TestBugCR15ListOpenFilesMultipartUpload.java
>
>
> h3. Mechanism
> {{OmMetadataManagerImpl.listOpenFiles}} takes the client ID of every listed
> row from its open key table key with
> {{OMMetadataManager.getClientIDFromOpenKeyDBKey}}, which runs
> {{Long.parseLong}} on the last path segment. The open key that
> {{S3InitiateMultipartUploadRequest}} writes for an upload ends with the
> upload ID instead ({{UUIDv7 + "-" + UniqueId}}, see
> {{OMMultipartUploadUtils.getMultipartUploadId}}), so the parse throws
> {{NumberFormatException}}. The exception is not an {{IOException}}, so
> {{OzoneManagerRequestHandler.handleReadRequest}} does not turn it into an
> error response. It reaches the client as
> {{RemoteException(java.lang.NumberFormatException)}}, and the whole listing
> fails, including the open keys before the upload.
> h3. Trigger
> # InitiateMultipartUpload in any bucket (FSO, OBJECT_STORE or LEGACY).
> # While the upload is open, list the open files of that bucket ({{ozone admin
> om lof -p /vol/bucket}}), or of {{/}} for an FSO bucket. Any page whose range
> contains the open key of the upload fails.
> No fault or race is needed. Every open S3 multipart upload has this row until
> it is completed or aborted.
> h3. Impact
> * The ListOpenFiles request, and with it {{ozone admin om list-open-files}},
> cannot list a bucket that has an open multipart upload, so an operator cannot
> see its open or hsync'ed files.
> * The client retries this error, so the command does not fail promptly. With
> default client settings on a single OM it had not returned after 5 minutes,
> and the OM logs a stack trace for every attempt.
> * No state is changed. The request is read only.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugCR15ListOpenFilesMultipartUpload.java]
> ({{hadoop-ozone/ozone-manager}}) starts one OM without datanodes and, for
> each bucket layout, opens one plain key and initiates one upload through
> {{OzoneManagerProtocol}}, then calls {{listOpenFiles}} on the bucket. The
> client retry settings are lowered so that the error reaches the caller. All
> three layouts fail with {{RemoteException(java.lang.NumberFormatException):
> For input string: "<upload ID>"}}, and the OM rejected both attempts. A
> passing test means the defect is present.
> h3. Patch
> [^CR-15-list-open-multipart-uploads-in-list-open-files.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 64d4a791502.
> {{listOpenFiles}} now recognises the open key of an upload by its key, with
> the existing {{OMMultipartUploadUtils.getUploadIdFromDbKey}}, and lists it
> with client ID 0 instead of parsing the upload ID, so {{ozone admin om lof}}
> shows it with client ID 0, which no real client has (client IDs come from
> {{UniqueId.next()}}). The key based check is used rather than the
> {{isMultipartKey}} flag because the open keys of parts being written also
> carry the flag and do end with a real client ID, and because upload rows
> written before HDDS-9017 do not carry the flag. Listing the row, rather than
> skipping it, keeps the listing consistent with {{totalOpenKeyCount}}, which
> already counts it, and leaves the pagination unchanged. Other rows are listed
> as before.
> Covered by {{testListOpenFilesWithMultipartUpload}} in the existing
> {{TestOmMetadataManager}}, for every bucket layout, with and without a
> continuation token. It lists a plain open key, an upload row without the
> {{isMultipartKey}} flag (as written before HDDS-9017) and the open key of a
> part being written, which has the flag and a real client ID. Without the
> change it fails with {{NumberFormatException}} for all layouts, and so would
> a check based on the flag. With it {{TestOmMetadataManager}} (33 tests) and
> the list open files tests of {{TestOzoneShellHA}} (2 tests, mini cluster)
> pass, and checkstyle is clean. With the patch the reproduction fails at
> {{assertThrows}}, because the listing succeeds.
> Found during review of a fix for the OM multipart upload complete path, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
