[
https://issues.apache.org/jira/browse/HDDS-17420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17420:
------------------------------
Attachment:
CR-10-return-not-implemented-for-if-none-match-etag-before-om-supports-it.patch
> S3 writes with an If-None-Match ETag return an unmapped HTTP 500, retried by
> SDKs, while OM does not support S3_IF_NONE_MATCH_ETAG
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17420
> URL: https://issues.apache.org/jira/browse/HDDS-17420
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-10-return-not-implemented-for-if-none-match-etag-before-om-supports-it.patch,
> TestBugCR10PreFinalizedIfNoneMatchS3Error.java
>
>
> h3. Mechanism
> When the OM version cached by {{RpcClient}} is below
> {{OzoneManagerVersion.S3_IF_NONE_MATCH_ETAG}} (104),
> {{RpcClient.checkIfNoneMatchETagSupport}} refuses {{createKeyIfNoneMatch}},
> {{createStreamKeyIfNoneMatch}} and {{completeMultipartUpload}} with an
> excluded ETag. It throws a plain {{IOException}}. S3 Gateway sends the
> condition from {{ObjectEndpoint.openKeyForPut}},
> {{ObjectEndpointStreaming.openStreamKeyForPut}} and
> {{ObjectEndpoint.completeMultipartUpload}}, and these paths translate only
> {{OMException}}. S3 Gateway has no exception mapper for {{IOException}}, so
> Jersey passes it to Jetty, which answers HTTP 500 without an S3 error
> document.
> Before HDDS-16622, S3 Gateway rejected a non {{*}} If-None-Match with 400
> InvalidRequest. The ZDU design says S3 Gateway returns clean errors when a
> feature is used before finalization.
> h3. Trigger
> # OM reports a version below 104 to clients, for example because it runs the
> current software but is not finalized, as during an upgrade before
> finalization.
> # An S3 client sends PutObject, CopyObject or CompleteMultipartUpload with
> {{If-None-Match: "<etag>"}}. {{If-None-Match: *}} is not affected.
> Because of HDDS-17140, S3 Gateway keeps the OM version it read first, so this
> continues after finalization until S3 Gateway restarts (from code, not run).
> h3. Impact
> * The client gets HTTP 500 with no S3 error code. PutObject and CopyObject
> return an empty body, and CompleteMultipartUpload returns a Jetty HTML page.
> * The AWS SDK v2 treats it as retryable and sends each request 4 times with
> its default retry policy.
> * S3 Gateway logs a WARN with a stack trace from
> {{HttpChannel.handleException}} on every attempt.
> The write is refused in every case, as intended. No data is affected.
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR10PreFinalizedIfNoneMatchS3Error.java]
> ({{hadoop-ozone/integration-test-s3}}) starts a mini cluster whose OM begins
> at apparent version ZDU (test only
> {{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}}), plus one S3 Gateway, and
> sends requests with the AWS SDK v2. A plain PutObject succeeds. With an
> If-None-Match ETag, PutObject, CopyObject and CompleteMultipartUpload each
> report {{HTTP 500, S3 error code null, SDK attempts 4}}, and S3 Gateway
> counts 4 failed requests for each. A passing test means the defect is
> present. A variant with datastream enabled
> ({{hdds.container.ratis.datastream.enabled}} and
> {{ozone.fs.datastream.enabled}} set to true,
> {{ozone.fs.datastream.auto.threshold}} 0MB), which sends PutObject and
> CopyObject through {{ObjectEndpointStreaming}}, gave the same result.
> h3. Patch
> [^CR-10-return-not-implemented-for-if-none-match-etag-before-om-supports-it.patch],
> against 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master
> at e03ba37c41d.
> {{RpcClient.checkIfNoneMatchETagSupport}} now throws {{OMException}} with
> {{NOT_SUPPORTED_OPERATION}} and the same message. {{OMException}} extends
> {{IOException}}, so no signature changes. All S3 Gateway paths above already
> translate it through {{S3ErrorTable}}, which maps {{NOT_SUPPORTED_OPERATION}}
> to 501 NotImplemented. The result is an S3 error document that the SDK does
> not retry. {{NOT_SUPPORTED_OPERATION}} is used instead of
> {{NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION}} because the client cannot tell
> a pre-finalized OM from older OM software. Finalized clusters are unaffected.
> Other version checks in {{RpcClient}} also throw a plain {{IOException}}, but
> they only trigger when OM runs older software than S3 Gateway, which the
> upgrade order rules out. They are not changed here.
> Covered by {{TestRpcClient.testIfNoneMatchETagReachesOMOrRejectsOldVersion}},
> which now also asserts the exception type and result code for every OM
> version below 104. Without the change, 21 of its 22 cases fail. With it:
> * the client unit tests (339) and S3 Gateway unit tests (1689) pass;
> * {{TestS3SDK}} on a mini cluster passes (249 tests, including the
> If-None-Match ETag cases on a finalized cluster);
> * checkstyle is clean.
> With the patch, the reproduction and its datastream variant fail because each
> request gets {{HTTP 501, S3 error code NotImplemented, SDK attempts 1}}.
> Related, not duplicates: HDDS-16622 (added the check), HDDS-17140 (S3 Gateway
> does not pick up the new OM version after ZDU finalization), HDDS-16994
> (wrong OM version check for atomic key creation).
> Found by code review of the zero downtime upgrade version gating in SCM, OM
> and the datanode write path, as part of the TLA+ verification effort under
> HDDS-15926, on commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
