[
https://issues.apache.org/jira/browse/HDDS-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17932876#comment-17932876
]
Ivan Andika edited comment on HDDS-12488 at 3/6/25 7:27 AM:
------------------------------------------------------------
[~adoroszlai] Thanks for checking this out.
Our cluster S3G uses https, I think x-amz-trailer is only added for https.
> Please also check HDDS-12123.
Thanks for the info. Might be related, but we don't encounter 500 error like in
HDDS-12123.
We had a user uploading a parquet file, but at the end of the uploaded key, it
will append "x-amz-checksum-crc32:ZS91yQ=="
I have added a possible fix in
https://github.com/ivandika3/ozone/tree/HDDS-12488, PTAL. I'm still trying to
write a test for this.
was (Author: JIRAUSER298977):
Our cluster S3G uses https, I think x-amz-trailer is only added for https.
> Please also check HDDS-12123.
Thanks for the info. Might be related, but we don't encounter 500 error like in
HDDS-12123.
We had a user uploading a parquet file, but at the end of the uploaded key, it
will append "x-amz-checksum-crc32:ZS91yQ=="
I have added a possible fix in
https://github.com/ivandika3/ozone/tree/HDDS-12488, PTAL. I'm still trying to
write a test for this.
> S3G should not append checksum trailer specified by x-amz-trailer
> -----------------------------------------------------------------
>
> Key: HDDS-12488
> URL: https://issues.apache.org/jira/browse/HDDS-12488
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We found that since version 1.36.0, Boto3 introduced a breaking change to
> their default integrity by using "x-amz-trailer" by default. This will make
> S3 PutObject and UploadPart requests to add a checksum trailer in the request
> body (e.g. "x-amz-checksum-crc32:ZS91yQ=="). Our S3G has not handled this so
> the checksum trailer will be appended in the request body, which is not
> expected.
> We need to handle this such that if "x-amz-trailer" is detected, we should
> not include x-amz-checksum-crc32" at the end.
> Resources:
> * https://github.com/boto/boto3/issues/4396
> * https://github.com/boto/boto3/issues/4392
> *
> https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
