CalvinKirs commented on code in PR #64695:
URL: https://github.com/apache/doris/pull/64695#discussion_r3577511477
##########
fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssObjStorage.java:
##########
@@ -473,6 +480,51 @@ private static String requireProperty(String value, String
key, String descripti
return value;
}
+ /**
+ * Computes the marker that resumes listing on the next page.
+ *
+ * <p>OSS (like the S3 V1 {@code GetBucket} API) only fills
+ * {@link ObjectListing#getNextMarker()} when a {@code delimiter} is set.
For a delimiter-less
+ * (recursive) listing the marker is {@code null} even when {@link
ObjectListing#isTruncated()}
+ * is {@code true}, so naively forwarding it would stop pagination after
the first page (~1000
+ * keys) and silently drop every object beyond it (recursive
delete/rename/list).
+ *
+ * <p>When the SDK leaves the marker blank on a truncated page, fall back
to the
+ * lexicographically greatest key/common-prefix returned on this page —
OSS lists in
+ * lexicographic order and resumes strictly after the marker. This mirrors
the last-key
+ * fallback the COS/OBS SDKs perform internally.
+ *
+ * <p>If a page is marked truncated yet yields neither a marker nor any
key/common-prefix to
+ * resume from, there is no cursor for the next page; returning {@code
null} would make the
+ * paginating caller re-list from the start indefinitely, so this fails
loudly instead.
+ */
+ private static String resolveNextMarker(ObjectListing listing) throws
IOException {
Review Comment:
Fixed in 0f5f7c1b622. `ObjectStorageFileIterator.hasNext()` now keeps
fetching while the filtered buffer is empty and the listing is not exhausted
(same do-while shape `listFiles()` already uses), so a truncated marker-only
page no longer terminates the iteration. Added `S3CompatibleFileSystemTest`
covering a marker-only truncated page followed by real files (also asserting
continuation-token propagation) and a marker-only final page. The fix lives in
the shared `S3CompatibleFileSystem`, so it covers s3/oss/obs/cos alike.
##########
fe/fe-filesystem/fe-filesystem-oss-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/OssHdfsFileSystemProvider.java:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.hdfs;
+
+import org.apache.doris.filesystem.FileSystem;
+import org.apache.doris.filesystem.hdfs.properties.OssHdfsProperties;
+import org.apache.doris.filesystem.properties.FileSystemProperties;
+import org.apache.doris.filesystem.spi.FileSystemProvider;
+
+import java.io.IOException;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * SPI provider for Aliyun OSS-HDFS (JindoFS), addressed with {@code oss://}
URIs and served by an
+ * HDFS-compatible {@code DFSFileSystem}. Registered via META-INF/services
alongside
+ * {@link HdfsFileSystemProvider}.
+ *
+ * <p>Routing is kept strictly disjoint from plain HDFS: the authoritative
{@code _STORAGE_TYPE_}
+ * marker set by fe-core's StoragePropertiesConverter wins, and the heuristic
fallback keys on
+ * {@link OssHdfsProperties#guessIsMe} (oss.hdfs flag / oss-dls endpoint) or
an {@code oss://} URI.
+ * The framework selects providers first-match-wins over an unordered list, so
the two providers'
+ * {@code supports()} predicates must not overlap.</p>
+ */
+public class OssHdfsFileSystemProvider implements
FileSystemProvider<FileSystemProperties> {
Review Comment:
Fixed in 54920958012. Added the same override as the sibling providers
(`ConnectorPropertiesUtils.getSensitiveKeys(OssHdfsProperties.class)`) plus a
unit test. This also closes a live gap:
`oss.hdfs.security_token`/`oss.security_token` were not masked anywhere before
(the legacy fe-core class does not mark them sensitive). The `fs.oss.*`
credential keys are covered too now that the plugin accepts them as binding
aliases (see the binder fix), and they were already registered via the
native-OSS provider in any case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
