This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new da848878439 [fix](fs) Restore legacy OSS bucket endpoint URI handling
(#69114)
da848878439 is described below
commit da8488784398c7176f0fb249847d9406c6e00825
Author: Gabriel <[email protected]>
AuthorDate: Fri Oct 16 09:34:05 2026 +0800
[fix](fs) Restore legacy OSS bucket endpoint URI handling (#69114)
### What problem does this PR solve?
Legacy OSS locations may encode the endpoint in the authority:
`scheme://<bucket>.<endpoint>/<path>`. FE-core normalizes these
locations for BE, but FE filesystem operations can receive the original
location retained in HMS.
Master now includes #68998, which handles the standard
`oss://<bucket>.oss-<region>.aliyuncs.com/<path>` form that triggered
the original `hive_on_hms_and_dlf` failure. The native OSS provider
still lacks the equivalent handling for its supported `s3`/`s3a` aliases
and custom endpoint authorities. For example, when the selected
filesystem provider is OSS:
```text
s3a://sample-bucket.oss-cn-hangzhou.aliyuncs.com/warehouse/part=value/file
oss://sample-bucket.custom-endpoint.example.com/warehouse/part=value/file
```
The remaining paths pass the entire authority as the bucket to the OSS
SDK, causing `The bucket name "..." is invalid`. They should instead use
bucket `sample-bucket`, preserve key `warehouse/part=value/file`, and
continue using the configured OSS endpoint. This PR restores the
provider-specific compatibility already implemented by FE-core's OSS
normalization.
Centralize the correction in the OSS provider and use it for all
URI-based object requests. Let the shared filesystem delegate URI
parsing to that provider as well: bulk/recursive deletion extracts
buckets before calling the SDK and must follow the same rule. Preserve
encoded keys and literal `+` characters; leave HTTP(S) parsing and other
providers unchanged, including valid dotted S3 bucket names.
### Release note
Fix Hive operations against legacy OSS locations containing a bucket and
endpoint in the URI authority.
### Check List (For Author)
- Test:
- [x] Unit Test: added seven focused tests. All seven tests pass on the
rebased branch, together with the upstream OSS regression tests.
- [x] `cd fe && mvn -pl fe-filesystem/fe-filesystem-oss -am test
-Dmaven.build.cache.enabled=false`: 369 tests passed after rebasing onto
master, including 51 OSS and 61 filesystem SPI tests. Maven Checkstyle
passed with zero violations.
- Coverage includes OSS/S3/S3A aliases under the OSS provider, both
addressing settings, encoded partition keys, custom endpoints and bucket
roots, write/copy/delete, bulk/recursive deletion, path-style HTTP,
unsupported schemes and dotted S3 buckets.
- `run-fe-ut.sh --run OssObjStorageTest` was attempted but stopped on a
local Thrift compiler mismatch (installed 0.16.0, required 0.24.0). The
focused modules above do not require Thrift generation. The full
cloud-backed Hive regression has not been rerun locally.
- Behavior changed:
- [x] Yes: native OSS requests accept legacy bucket.endpoint locations
consistently with existing FE path normalization.
- Does this need documentation?
- [x] No: restores existing URI compatibility without adding
configuration.
---
.../apache/doris/filesystem/oss/OssFileSystem.java | 6 ++
.../apache/doris/filesystem/oss/OssObjStorage.java | 39 +++++--
.../doris/filesystem/oss/OssObjStorageTest.java | 116 +++++++++++++++++++++
.../filesystem/spi/S3CompatibleFileSystem.java | 8 +-
.../doris/filesystem/spi/ObjectStorageUriTest.java | 8 ++
5 files changed, 164 insertions(+), 13 deletions(-)
diff --git
a/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssFileSystem.java
b/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssFileSystem.java
index dd29d9b729e..7758054dbfb 100644
---
a/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssFileSystem.java
+++
b/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssFileSystem.java
@@ -17,6 +17,7 @@
package org.apache.doris.filesystem.oss;
+import org.apache.doris.filesystem.spi.ObjectStorageUri;
import org.apache.doris.filesystem.spi.S3CompatibleFileSystem;
/**
@@ -27,4 +28,9 @@ public class OssFileSystem extends S3CompatibleFileSystem {
public OssFileSystem(OssObjStorage objStorage) {
super(objStorage, objStorage.isUsePathStyle(),
objStorage.getSupportedSchemes());
}
+
+ @Override
+ protected ObjectStorageUri parseUri(String uri) {
+ return ((OssObjStorage) objStorage).parseUri(uri);
+ }
}
diff --git
a/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssObjStorage.java
b/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssObjStorage.java
index cfb1da68532..b17c21f114f 100644
---
a/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssObjStorage.java
+++
b/fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssObjStorage.java
@@ -115,6 +115,23 @@ public class OssObjStorage implements ObjStorage<OSS> {
return properties.getSupportedSchemes();
}
+ ObjectStorageUri parseUri(String path) {
+ ObjectStorageUri uri = ObjectStorageUri.parse(path, isUsePathStyle(),
getSupportedSchemes());
+ if (path.regionMatches(true, 0, "http://";, 0, 7)
+ || path.regionMatches(true, 0, "https://";, 0, 8)) {
+ return uri;
+ }
+ int dot = uri.bucket().indexOf('.');
+ if (dot <= 0) {
+ return uri;
+ }
+ // HMS can retain legacy bucket.endpoint locations even when BE paths
are normalized.
+ // Rewrite only in the OSS provider: S3 bucket names may contain dots.
Preserve raw keys
+ // so encoded partition separators and literal '+' characters still
identify the same objects.
+ return ObjectStorageUri.parse("oss://" + uri.bucket().substring(0,
dot) + "/" + uri.key(),
+ isUsePathStyle(), getSupportedSchemes());
+ }
+
@Override
public OSS getClient() throws IOException {
return getClient(properties.getBucket());
@@ -199,7 +216,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public RemoteObjects listObjectsWithOptions(String remotePath,
ObjectListOptions options) throws IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
ListObjectsRequest request = new ListObjectsRequest(uri.bucket());
request.setPrefix(uri.key());
if (options != null) {
@@ -233,7 +250,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public RemoteObject headObject(String remotePath) throws IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
try {
ObjectMetadata metadata =
getClient(uri.bucket()).getObjectMetadata(uri.bucket(), uri.key());
return new RemoteObject(uri.key(), uri.key(), metadata.getETag(),
metadata.getContentLength(),
@@ -250,7 +267,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public void putObject(String remotePath, RequestBody requestBody) throws
IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(requestBody.contentLength());
try (InputStream content = requestBody.content()) {
@@ -264,7 +281,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public void deleteObject(String remotePath) throws IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
try {
getClient(uri.bucket()).deleteObject(uri.bucket(), uri.key());
} catch (OSSException e) {
@@ -279,8 +296,8 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public void copyObject(String srcPath, String dstPath) throws IOException {
- ObjectStorageUri src = ObjectStorageUri.parse(srcPath,
isUsePathStyle(), getSupportedSchemes());
- ObjectStorageUri dst = ObjectStorageUri.parse(dstPath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri src = parseUri(srcPath);
+ ObjectStorageUri dst = parseUri(dstPath);
try {
getClient(src.bucket()).copyObject(new CopyObjectRequest(
src.bucket(), src.key(), dst.bucket(), dst.key()));
@@ -292,7 +309,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public String initiateMultipartUpload(String remotePath) throws
IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
try {
InitiateMultipartUploadResult result =
getClient(uri.bucket()).initiateMultipartUpload(
new InitiateMultipartUploadRequest(uri.bucket(),
uri.key()));
@@ -306,7 +323,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public UploadPartResult uploadPart(String remotePath, String uploadId, int
partNum,
RequestBody body) throws IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
try (InputStream content = body.content()) {
com.aliyun.oss.model.UploadPartRequest request = new
com.aliyun.oss.model.UploadPartRequest();
request.setBucketName(uri.bucket());
@@ -326,7 +343,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public void completeMultipartUpload(String remotePath, String uploadId,
List<UploadPartResult> parts) throws IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
List<PartETag> partEtags = parts.stream()
.map(part -> new PartETag(part.partNumber(), part.etag()))
.collect(Collectors.toList());
@@ -341,7 +358,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public void abortMultipartUpload(String remotePath, String uploadId)
throws IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
try {
getClient(uri.bucket()).abortMultipartUpload(new
AbortMultipartUploadRequest(
uri.bucket(), uri.key(), uploadId));
@@ -353,7 +370,7 @@ public class OssObjStorage implements ObjStorage<OSS> {
@Override
public InputStream openInputStreamAt(String remotePath, long fromByte)
throws IOException {
- ObjectStorageUri uri = ObjectStorageUri.parse(remotePath,
isUsePathStyle(), getSupportedSchemes());
+ ObjectStorageUri uri = parseUri(remotePath);
try {
GetObjectRequest request = new GetObjectRequest(uri.bucket(),
uri.key());
if (fromByte > 0) {
diff --git
a/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssObjStorageTest.java
b/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssObjStorageTest.java
index e557875cbeb..60fa7a0972a 100644
---
a/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssObjStorageTest.java
+++
b/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssObjStorageTest.java
@@ -197,6 +197,122 @@ class OssObjStorageTest {
Assertions.assertFalse(request.getHeaders().containsKey(OSSHeaders.OSS_SECURITY_TOKEN));
}
+ @Test
+ void listObjects_acceptsLegacyBucketEndpointWithoutDecodingKeys() throws
Exception {
+ // HMS retains the original table location even when BE receives a
normalized write path.
+ String key = "warehouse/pt=a%2Fb%252Fc+value/";
+ for (String scheme : List.of("oss", "s3", "s3a")) {
+ for (boolean pathStyle : List.of(false, true)) {
+ OSS mockOss = Mockito.mock(OSS.class);
+
Mockito.when(mockOss.listObjects(Mockito.any(ListObjectsRequest.class)))
+ .thenReturn(new ObjectListing());
+ Map<String, String> props = buildBasicProps();
+ props.put("use_path_style", Boolean.toString(pathStyle));
+ OssObjStorage storage = new TestableOssObjStorage(props,
mockOss);
+
+ storage.listObjects(scheme +
"://my-bucket.oss-cn-hangzhou.aliyuncs.com/" + key, null);
+
+ ArgumentCaptor<ListObjectsRequest> captor =
ArgumentCaptor.forClass(ListObjectsRequest.class);
+ Mockito.verify(mockOss).listObjects(captor.capture());
+ Assertions.assertEquals("my-bucket",
captor.getValue().getBucketName());
+ Assertions.assertEquals(key, captor.getValue().getPrefix());
+ }
+ }
+ }
+
+ @Test
+ void headObject_acceptsBucketEndpointAndBucketRoot() throws Exception {
+ OSS mockOss = Mockito.mock(OSS.class);
+ Mockito.when(mockOss.getObjectMetadata(Mockito.anyString(),
Mockito.anyString()))
+ .thenReturn(new ObjectMetadata());
+ OssObjStorage storage = new TestableOssObjStorage(buildBasicProps(),
mockOss);
+
+
storage.headObject("oss://my-bucket.custom-endpoint.example.com/dir/a%2Fb+file");
+ storage.headObject("oss://my-bucket.oss-cn-hangzhou.aliyuncs.com");
+
+ Mockito.verify(mockOss).getObjectMetadata("my-bucket",
"dir/a%2Fb+file");
+ Mockito.verify(mockOss).getObjectMetadata("my-bucket", "");
+ }
+
+ @Test
+ void deleteFiles_usesSameBucketParsingAsObjectRequests() throws Exception {
+ // Batch deletion extracts buckets in the shared filesystem, bypassing
URI-based SDK methods.
+ OSS mockOss = Mockito.mock(OSS.class);
+
Mockito.when(mockOss.deleteObjects(Mockito.any(DeleteObjectsRequest.class)))
+ .thenReturn(new DeleteObjectsResult(Collections.emptyList()));
+ OssFileSystem filesystem = new OssFileSystem(new
TestableOssObjStorage(buildBasicProps(), mockOss));
+
+ filesystem.deleteFiles(List.of(
+
Location.of("oss://my-bucket.oss-cn-hangzhou.aliyuncs.com/dir/a%2Fb"),
+ Location.of("s3://my-bucket/dir/c+d")));
+
+ ArgumentCaptor<DeleteObjectsRequest> captor =
ArgumentCaptor.forClass(DeleteObjectsRequest.class);
+ Mockito.verify(mockOss).deleteObjects(captor.capture());
+ Assertions.assertEquals("my-bucket",
captor.getValue().getBucketName());
+ Assertions.assertEquals(List.of("dir/a%2Fb", "dir/c+d"),
captor.getValue().getKeys());
+ }
+
+ @Test
+ void writeCopyAndDelete_acceptLegacyAuthorities() throws Exception {
+ OSS mockOss = Mockito.mock(OSS.class);
+ OssObjStorage storage = new TestableOssObjStorage(buildBasicProps(),
mockOss);
+ String source =
"oss://my-bucket.oss-cn-hangzhou.aliyuncs.com/dir/a%2Fb";
+ String target =
"s3://other-bucket.oss-cn-hangzhou.aliyuncs.com/dir/a+b";
+
+ storage.putObject(source, RequestBody.of(new ByteArrayInputStream(new
byte[] {1}), 1));
+ storage.copyObject(source, target);
+ storage.deleteObject(source);
+
+ ArgumentCaptor<PutObjectRequest> put =
ArgumentCaptor.forClass(PutObjectRequest.class);
+ Mockito.verify(mockOss).putObject(put.capture());
+ Assertions.assertEquals("my-bucket", put.getValue().getBucketName());
+ Assertions.assertEquals("dir/a%2Fb", put.getValue().getKey());
+ ArgumentCaptor<CopyObjectRequest> copy =
ArgumentCaptor.forClass(CopyObjectRequest.class);
+ Mockito.verify(mockOss).copyObject(copy.capture());
+ Assertions.assertEquals("my-bucket",
copy.getValue().getSourceBucketName());
+ Assertions.assertEquals("dir/a%2Fb", copy.getValue().getSourceKey());
+ Assertions.assertEquals("other-bucket",
copy.getValue().getDestinationBucketName());
+ Assertions.assertEquals("dir/a+b",
copy.getValue().getDestinationKey());
+ Mockito.verify(mockOss).deleteObject("my-bucket", "dir/a%2Fb");
+ }
+
+ @Test
+ void recursiveDelete_usesNormalizedBucketForBatchRequest() throws
Exception {
+ OSS mockOss = Mockito.mock(OSS.class);
+ OSSObjectSummary summary = new OSSObjectSummary();
+ summary.setKey("dir/pt=a%2Fb/file");
+ ObjectListing listing = new ObjectListing();
+ listing.setObjectSummaries(List.of(summary));
+
Mockito.when(mockOss.listObjects(Mockito.any(ListObjectsRequest.class))).thenReturn(listing);
+
Mockito.when(mockOss.deleteObjects(Mockito.any(DeleteObjectsRequest.class)))
+ .thenReturn(new DeleteObjectsResult(Collections.emptyList()));
+ OssFileSystem filesystem = new OssFileSystem(new
TestableOssObjStorage(buildBasicProps(), mockOss));
+
+
filesystem.delete(Location.of("oss://my-bucket.oss-cn-hangzhou.aliyuncs.com/dir/"),
true);
+
+ ArgumentCaptor<DeleteObjectsRequest> captor =
ArgumentCaptor.forClass(DeleteObjectsRequest.class);
+ Mockito.verify(mockOss).deleteObjects(captor.capture());
+ Assertions.assertEquals("my-bucket",
captor.getValue().getBucketName());
+ Assertions.assertEquals(List.of("dir/pt=a%2Fb/file"),
captor.getValue().getKeys());
+ }
+
+ @Test
+ void headObject_keepsPathStyleHttpBucketAndRejectsUnsupportedSchemes()
throws Exception {
+ OSS mockOss = Mockito.mock(OSS.class);
+ Mockito.when(mockOss.getObjectMetadata(Mockito.anyString(),
Mockito.anyString()))
+ .thenReturn(new ObjectMetadata());
+ Map<String, String> props = buildBasicProps();
+ props.put("use_path_style", "true");
+ OssObjStorage storage = new TestableOssObjStorage(props, mockOss);
+
+
storage.headObject("https://oss-cn-hangzhou.aliyuncs.com/my-bucket/dir/a%2Fb";);
+
+ Mockito.verify(mockOss).getObjectMetadata("my-bucket", "dir/a%2Fb");
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () ->
storage.headObject("cos://my-bucket.endpoint.example.com/key"));
+ Mockito.verifyNoMoreInteractions(mockOss);
+ }
+
@Test
void listObjects_usesOssNativeClientAndMapsResult() throws Exception {
OSS mockOss = Mockito.mock(OSS.class);
diff --git
a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/S3CompatibleFileSystem.java
b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/S3CompatibleFileSystem.java
index 5942aa95a37..93fc8c0ba8d 100644
---
a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/S3CompatibleFileSystem.java
+++
b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/S3CompatibleFileSystem.java
@@ -210,8 +210,12 @@ public abstract class S3CompatibleFileSystem extends
ObjFileSystem {
} while (continuationToken != null);
}
- /** Parses {@code uri} respecting the underlying client's path-style
configuration. */
- private ObjectStorageUri parseUri(String uri) {
+ /**
+ * Parses {@code uri} respecting the underlying client's path-style
configuration.
+ * Providers must apply the same bucket parsing here and in object
requests, since batch
+ * deletion extracts bucket names here before calling the native client.
+ */
+ protected ObjectStorageUri parseUri(String uri) {
return ObjectStorageUri.parse(uri, usePathStyle, supportedSchemes);
}
diff --git
a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/ObjectStorageUriTest.java
b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/ObjectStorageUriTest.java
index 3c2557070da..21fe4a6842b 100644
---
a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/ObjectStorageUriTest.java
+++
b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/ObjectStorageUriTest.java
@@ -62,6 +62,14 @@ class ObjectStorageUriTest {
Assertions.assertEquals("path/to/file.csv", uri.key());
}
+ @Test
+ void parsePreservesDottedBucketsOutsideOssProvider() {
+ // OSS's legacy bucket.endpoint rewrite must not change legal dotted
S3 bucket names.
+ ObjectStorageUri uri =
ObjectStorageUri.parse("s3://logs.prod/dir/a%2Fb+file", false);
+ Assertions.assertEquals("logs.prod", uri.bucket());
+ Assertions.assertEquals("dir/a%2Fb+file", uri.key());
+ }
+
@Test
void parseS3aScheme() {
ObjectStorageUri uri = ObjectStorageUri.parse("s3a://my-bucket/key",
false);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
