0AyanamiRei opened a new pull request, #65504:
URL: https://github.com/apache/doris/pull/65504
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #63409
Problem Summary:
Doris cannot currently read from or write to Amazon S3 Express One Zone
directory buckets.
There are two compatibility gaps:
1. The bundled AWS SDK for C++ is 1.11.219 and the S3 client is constructed
through the legacy client configuration path. This does not enable the
endpoint-rules resolver required for S3 Express directory buckets, including
bucket-specific endpoint resolution, `CreateSession` authentication, and the
`s3express` signing service.
2. Doris always sends `Content-MD5` for `PutObject` and `UploadPart`. S3
Express One Zone does not support `Content-MD5` for these requests and requires
a flexible checksum such as CRC32C.
This PR takes over and supersedes #63409. It:
- Upgrades aws-sdk-cpp from 1.11.219 to 1.11.400.
- Creates the S3 client with `Aws::S3::S3ClientConfiguration` and
`S3EndpointProvider` so AWS endpoint rules and S3 Express session
authentication are enabled.
- Uses CRC32C rather than `Content-MD5` for S3 Express uploads, while
preserving the existing MD5 behavior for regular S3 and S3-compatible endpoints.
- Adds `s3_disable_content_md5` as an explicit compatibility switch for
endpoints that do not support `Content-MD5`.
- Centralizes S3 Express context detection and addresses the review comments
on #63409: directory buckets must end with `--x-s3`, and endpoints must contain
an S3 Express hostname label rather than an arbitrary substring.
- Adds unit coverage for valid directory bucket names, valid S3 Express
endpoints, and false-positive bucket/endpoint cases.
### Release note
Support reading from and writing to Amazon S3 Express One Zone directory
buckets. S3 Express uploads use CRC32C instead of `Content-MD5`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- Added `S3UTILTest.is_s3_express_context`.
- Local ASAN UT cold build was started after repairing a broken
`be/ut_build_ASAN` symlink, but was interrupted at 13056/15380 build targets
before the test binary was produced. The PR is opened as draft until the unit
test and CI complete.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes.
- S3 Express directory buckets use SDK endpoint resolution and
session authentication.
- `PutObject` and `UploadPart` use CRC32C instead of `Content-MD5`
in the S3 Express context.
- Regular S3 behavior remains unchanged by default.
- Does this need documentation?
- [ ] No.
- [x] Yes. A follow-up doris-website PR should document S3 Express
directory bucket configuration and `s3_disable_content_md5`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
