hubgeter opened a new pull request, #68879:
URL: https://github.com/apache/doris/pull/68879
### What problem does this PR solve?
Issue Number: N/A
Related PR: #64695, #66004, #58099
Problem Summary:
Object storage endpoints are DNS host names, but the storage-type routing
and the endpoint -> region parsing compared them with case-sensitive String
operations. A semantically identical but differently cased endpoint therefore
behaved differently from its lowercase spelling:
- oss.endpoint = CN-BEIJING.OSS-DLS.ALIYUNCS.COM matched no
FileSystemProvider at all ("No supported storage type found"), while the
lowercase spelling was routed to OSS-HDFS.
- The same held for native OSS (.aliyuncs.com), S3 (amazonaws.com), COS
(myqcloud.com) and OBS (myhuaweicloud.com).
- The region parsing in S3URI / LegacyS3Uri / S3ResourceCompat used
endpointSplits[0].contains("oss-"), so OSS-CN-BEIJING.aliyuncs.com yielded the
bogus region "ALIYUNCS"; S3ResourceCompat.getRegionOfEndpoint additionally
stripped the scheme with a case-sensitive replace("https://";, "").
Fix: match endpoint hosts case-insensitively (the oss- region marker
included) and normalise the effective OSS endpoint to lowercase. Normalising is
not cosmetic: OssFileSystemProperties hands the effective endpoint to the BE as
AWS_ENDPOINT, and S3FileSystem::generate_presigned_url matches
"-internal.aliyuncs.com" case-sensitively when it converts an internal endpoint
into the public one.
Note for reviewers: fe-filesystem is a behaviour-preserving port of the
deleted fe-core storage properties (several sites still carry "byte-identical
to legacy" / "Verbatim port" comments). The case-sensitive comparisons were
reproduced deliberately by that port, so this PR is an intentional deviation
from the legacy behaviour and the comments at the touched sites were updated
accordingly. All-lowercase configurations are unaffected: for a lowercase input
containsIgnoreCase == contains, toLowerCase(ROOT).endsWith == endsWith, and
lowercasing is idempotent.
Validation: full unit-test suites of the seven touched fe-filesystem
modules, 121 fe-core storage/routing tests (including the frozen
bindPrimary/bindAll routing contract and the routing parity tests), Checkstyle
on all touched modules and git diff --check, all green. A smoke test against a
real OSS-DLS endpoint (JindoFS downstream and the BE file-type switch) is still
pending and will be reported here.
### Release note
Make object storage endpoint matching case-insensitive (OSS/OSS-HDFS, S3,
COS, OBS, plus the endpoint -> region parsing) and normalise the effective OSS
endpoint to lowercase.
Note for configurations that declare OSS explicitly (fs.oss.support=true):
an upper/mixed case spelling of a standard public OSS endpoint
(oss-<region>.aliyuncs.com) is no longer silently rewritten to the internal
endpoint (oss-<region>-internal.aliyuncs.com). It now uses the public endpoint,
exactly like the lowercase spelling did. To keep using the internal endpoint,
configure it explicitly, e.g. oss.endpoint = oss-<region>-internal.aliyuncs.com.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
