[
https://issues.apache.org/jira/browse/HDDS-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuo Huang updated HDDS-16672:
------------------------------
Description:
Problem
The ceph/s3-tests test_object_set_get_unicode_metadata test fails against Ozone
when an x-amz-meta-* value contains non-ASCII characters.
For example:
x-amz-meta-meta1: Hello Worldé
is not returned correctly by S3 Gateway.
Investigation showed that Amazon S3 uses RFC 2047 for non-ASCII user metadata.
Ozone currently does not decode RFC 2047 request values or encode non-ASCII
response values.
Expected behavior
- Decode RFC 2047 Q/B values before storing user metadata.
- Encode non-ASCII metadata in GET and HEAD responses.
- Return ASCII metadata unchanged.
References
ceph/s3-tests:
https://github.com/ceph/s3-tests/blob/fb8b73092bb1dd8db829f1205a9e52e73bf9a232/s3tests/functional/test_s3.py#L1883-L1899
Amazon S3 metadata behavior:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata
was:
h3. Problem
S3 Gateway corrupts non-ASCII characters in user-defined object metadata.
For example, an object uploaded with:
{{x-amz-meta-meta1: Hello Worldé}}
returns:
{{x-amz-meta-meta1: Hello Worldé}}
AWS S3 supports Unicode characters in user-defined metadata values.
h3. Reproduction
Start an Ozone cluster and configure ceph/s3-tests to use its S3 endpoint, then
run:
{{S3TEST_CONF=<path-to-s3tests.conf> python -m pytest -v
s3tests/functional/test_s3.py::test_object_set_get_unicode_metadata}}
h3. Actual result
The test fails with:
{{AssertionError: assert 'Hello Worldé' == 'Hello Worldé'}}
h3. Expected result
The metadata value returned by GetObject should match the uploaded value:
{{Hello Worldé}}
h3. Root cause
The UTF-8 header bytes are exposed to S3 Gateway as a byte-preserving Latin-1
string. {{EndpointBase.getCustomMetadataFromHeaders()}} stores that string
without decoding it back to UTF-8.
h3. References
* [AWS S3 user-defined
metadata|https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html]
* [ceph/s3-tests
reproduction|https://github.com/ceph/s3-tests/blob/fb8b73092bb1dd8db829f1205a9e52e73bf9a232/s3tests/functional/test_s3.py#L1885-L1899]
* [Ozone S3 compatibility report|https://ozone.s3.peterxcli.dev/]
> S3 Gateway corrupts Unicode user metadata values
> ------------------------------------------------
>
> Key: HDDS-16672
> URL: https://issues.apache.org/jira/browse/HDDS-16672
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Shuo Huang
> Assignee: Shuo Huang
> Priority: Major
> Labels: pull-request-available
>
> Problem
> The ceph/s3-tests test_object_set_get_unicode_metadata test fails against
> Ozone when an x-amz-meta-* value contains non-ASCII characters.
> For example:
> x-amz-meta-meta1: Hello Worldé
> is not returned correctly by S3 Gateway.
> Investigation showed that Amazon S3 uses RFC 2047 for non-ASCII user
> metadata. Ozone currently does not decode RFC 2047 request values or encode
> non-ASCII response values.
> Expected behavior
> - Decode RFC 2047 Q/B values before storing user metadata.
> - Encode non-ASCII metadata in GET and HEAD responses.
> - Return ASCII metadata unchanged.
> References
> ceph/s3-tests:
> https://github.com/ceph/s3-tests/blob/fb8b73092bb1dd8db829f1205a9e52e73bf9a232/s3tests/functional/test_s3.py#L1883-L1899
> Amazon S3 metadata behavior:
> https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
