github-actions[bot] commented on code in PR #70443:
URL: https://github.com/apache/doris/pull/70443#discussion_r4597719869


##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -1909,42 +1907,21 @@ Status 
CloudMetaMgr::get_storage_vault_info(StorageVaultInfos* vault_infos, bool
 
     *is_vault_mode = resp.enable_storage_vault();
 
-    auto add_obj_store = [&vault_infos](const auto& obj_store) {
-        vault_infos->emplace_back(obj_store.id(), 
S3Conf::get_s3_conf(obj_store),
-                                  StorageVaultPB_PathFormat {});
-    };
-
-    std::ranges::for_each(resp.obj_info(), add_obj_store);
-    std::ranges::for_each(resp.storage_vault(), [&](const auto& vault) {
+    for (const auto& info : resp.obj_info()) {
+        vault_infos->emplace_back(info.id(), info, StorageVaultPB_PathFormat 
{});
+    }
+    for (const auto& vault : resp.storage_vault()) {
         if (vault.has_hdfs_info()) {
             vault_infos->emplace_back(vault.id(), vault.hdfs_info(), 
vault.path_format());
+        } else if (vault.has_obj_info()) {
+            const auto& id = vault.id().empty() ? vault.obj_info().id() : 
vault.id();
+            vault_infos->emplace_back(id, vault.obj_info(), 
vault.path_format());

Review Comment:
   [P1] Preserve the legacy object-vault layout during upgrade
   
   An operator-provisioned built-in S3 vault can persist 
`path_format={path_version:1,shard_num:1000}` through `create_instance(vault)` 
-> `add_vault_into_instance`. Older BEs ignored that field for every object 
vault and wrote layout 0. Forwarding it here makes an upgraded BE construct 
`StorageResource` with layout 1, so `Rowset::segment_path` looks for 
`data/<shard>/<tablet>/<rowset>/<segment>.dat` instead of the existing 
`data/<tablet>/<rowset>_<segment>.dat`. Existing committed rowsets become 
unreadable after restart/upgrade, and mixed-version BEs disagree about new 
paths. Preserve the historical S3 layout in this refactor or introduce an 
explicit compatible migration/layout selector.



##########
common/cpp/obj-storage/client/s3_obj_storage_client.cpp:
##########
@@ -117,6 +191,9 @@ ObjStorageUploadResult 
S3ObjStorageClient::create_multipart_upload(const ObjStor
 ObjStorageResponse S3ObjStorageClient::put_object(const ObjStoragePath& opts,
                                                   std::string_view stream) {
     Aws::S3::Model::PutObjectRequest request;
+    if (_supports_s3_encryption && opts.sse_enabled) {
+        
request.SetServerSideEncryption(Aws::S3::Model::ServerSideEncryption::AES256);

Review Comment:
   [P1] Avoid enabling SSE on existing MinIO vaults by default
   
   Built-in object vaults already have `sse_enabled=true` because 
`add_vault_into_instance` hardcodes it, even when `create_instance` has no SSE 
opt-in. The repository's `docker/runtime/all-in-one/4.1/resource/cloud_init.sh` 
uses that default vault mode, and its `compose/cloud.yml` starts MinIO without 
KMS. The newly propagated flag now adds `AES256` here and during multipart 
initiation; MinIO [rejects SSE-S3 without configured 
KMS](https://github.com/minio/minio/blob/RELEASE.2025-01-20T14-49-07Z/cmd/encryption-v1.go#L363-L369),
 so previously working built-in MinIO vaults can no longer accept data writes. 
Preserve prior upload behavior unless encryption was explicitly enabled and 
supported, rather than turning this historically forced metadata flag into a 
new storage requirement.



##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -456,6 +448,9 @@ Status 
CloudStorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sp
 }
 
 void CloudStorageEngine::sync_storage_vault() {
+    // The periodic refresh and a lookup for a missing vault can overlap. 
Serialize the fetch as
+    // well as publication, so an older metadata snapshot cannot overwrite 
newer credentials.
+    std::lock_guard lock(_sync_storage_vault_mutex);

Review Comment:
   [P1] Use a bthread-aware mutex for vault refresh
   
   Missing-vault lookups run in `CloudDeltaWriter::batch_init`'s 
`bthread_fork_join` tasks (`writer->init` -> `CloudRowsetBuilder::init` -> 
`get_storage_resource`). This `std::mutex` remains locked while 
`get_storage_vault_info` waits for its synchronous RPC, including `retry_rpc`'s 
`bthread_usleep`. When enough tablets initialize against a newly created vault 
concurrently, mutex waiters can park all bthread worker pthreads while the 
owner is suspended waiting for RPC/retry completion; it then cannot resume to 
release the lock, hanging refresh and loads. Keep serialization with 
`bthread::Mutex` or an ordinary-thread refresh executor.



##########
common/cpp/obj-storage/parser/obj_storage_config_parser.cpp:
##########
@@ -0,0 +1,686 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cpp/obj-storage/parser/obj_storage_config_parser.h"
+
+#include <brpc/uri.h>
+#include <fmt/format.h>
+#include <gen_cpp/AgentService_types.h>
+
+#include <algorithm>
+#include <cctype>
+#include <charconv>
+#include <type_traits>
+#include <utility>
+
+#include "cpp/obj-storage/auth/gcp/gcp_auth.h"
+
+namespace doris {
+namespace {
+using Vault = cloud::ObjectStoreInfoPB;
+
+struct ProviderMapping {
+    std::string_view name;
+    TObjStorageType::type resource;
+    cloud::ObjectStoreInfoPB::Provider vault;
+};
+
+constexpr ProviderMapping PROVIDERS[] = {
+        {"AWS", TObjStorageType::AWS, cloud::ObjectStoreInfoPB::S3},
+        {"S3", TObjStorageType::AWS, cloud::ObjectStoreInfoPB::S3},
+        {"OSS", TObjStorageType::OSS, cloud::ObjectStoreInfoPB::OSS},
+        {"COS", TObjStorageType::COS, cloud::ObjectStoreInfoPB::COS},
+        {"BOS", TObjStorageType::BOS, cloud::ObjectStoreInfoPB::BOS},
+        {"OBS", TObjStorageType::OBS, cloud::ObjectStoreInfoPB::OBS},
+        {"GCP", TObjStorageType::GCP, cloud::ObjectStoreInfoPB::GCP},
+        {"GCS", TObjStorageType::GCP, cloud::ObjectStoreInfoPB::GCP},
+        {"TOS", TObjStorageType::TOS, cloud::ObjectStoreInfoPB::TOS},
+        {"AZURE", TObjStorageType::AZURE, cloud::ObjectStoreInfoPB::AZURE},
+};
+
+std::optional<std::string_view> 
credential_provider_name(TCredProviderType::type provider) {
+    switch (provider) {
+    case TCredProviderType::DEFAULT:
+        return "DEFAULT";
+    case TCredProviderType::SIMPLE:
+        return "SIMPLE";
+    case TCredProviderType::INSTANCE_PROFILE:
+        return "INSTANCE_PROFILE";
+    case TCredProviderType::ENV:
+        return "ENV";
+    case TCredProviderType::SYSTEM_PROPERTIES:
+        return "SYSTEM_PROPERTIES";
+    case TCredProviderType::WEB_IDENTITY:
+        return "WEB_IDENTITY";
+    case TCredProviderType::CONTAINER:
+        return "CONTAINER";
+    case TCredProviderType::ANONYMOUS:
+        return "ANONYMOUS";
+    default:
+        return std::nullopt;
+    }
+}
+
+std::optional<std::string_view> gcp_credential_provider_name(
+        TGcpCredentialProviderType::type provider) {
+    switch (provider) {
+    case TGcpCredentialProviderType::DEFAULT:
+        return "DEFAULT";
+    case TGcpCredentialProviderType::COMPUTE_ENGINE:
+        return "COMPUTE_ENGINE";
+    default:
+        return std::nullopt;
+    }
+}
+
+std::string to_upper(std::string text) {
+    std::transform(text.begin(), text.end(), text.begin(),
+                   [](unsigned char c) { return 
static_cast<char>(std::toupper(c)); });
+    return text;
+}
+
+std::string normalize_provider(std::string text) {
+    const auto non_space = [](unsigned char c) { return !std::isspace(c); };
+    text.erase(text.begin(), std::find_if(text.begin(), text.end(), 
non_space));
+    text.erase(std::find_if(text.rbegin(), text.rend(), non_space).base(), 
text.end());
+    return to_upper(std::move(text));
+}
+
+std::string_view pop_dns_label(std::string_view& name) {
+    const auto dot = name.rfind('.');
+    if (dot == std::string_view::npos) {
+        return std::exchange(name, std::string_view {});
+    }
+    auto label = name.substr(dot + 1);
+    name = name.substr(0, dot);
+    return label;
+}
+
+} // namespace
+
+cloud::ObjectStoreInfoPB::Provider vault_storage_provider(const 
cloud::ObjectStoreInfoPB& info) {
+    if (info.has_provider() || info.endpoint().empty()) {
+        return info.provider();
+    }
+    return storage_provider_from_endpoint(info.endpoint());
+}
+
+cloud::ObjectStoreInfoPB::Provider 
storage_provider_from_endpoint(std::string_view endpoint) {
+    std::string url(endpoint);
+    std::transform(url.begin(), url.end(), url.begin(),
+                   [](unsigned char c) { return std::tolower(c); });
+    brpc::URI uri;
+    if (url.find('\0') != std::string::npos || uri.SetHttpURL(url.c_str()) != 
0 ||
+        (!uri.scheme().empty() && uri.scheme() != "http" && uri.scheme() != 
"https")) {
+        return Vault::S3;
+    }
+    std::string_view host = uri.host();
+    if (host.ends_with('.')) {
+        host.remove_suffix(1);
+    }
+    if (host.empty() || 
host.find_first_not_of("abcdefghijklmnopqrstuvwxyz0123456789-.") !=
+                                std::string_view::npos) {
+        return Vault::S3;
+    }
+
+    // Match from the DNS root so virtual-hosted buckets cannot change the 
service or cloud.
+    auto labels = host;
+    auto tld = pop_dns_label(labels);
+    auto domain = pop_dns_label(labels);
+    auto last = pop_dns_label(labels);
+    auto previous = pop_dns_label(labels);
+    auto is_region = [](std::string_view name) {
+        return name.size() > 3 && name[2] == '-' && name[0] >= 'a' && name[0] 
<= 'z' &&
+               name[1] >= 'a' && name[1] <= 'z';
+    };
+    if (domain == "aliyuncs" && tld == "com") {
+        if ((last.starts_with("oss-") && is_region(last.substr(4))) || last == 
"oss-accelerate" ||
+            last == "oss-accelerate-overseas" || (previous == "s3" && 
is_region(last)) ||
+            (last == "oss" && is_region(previous))) {
+            return Vault::OSS;
+        }
+    } else if (domain == "googleapis" && tld == "com" && last == "storage") {
+        return Vault::GCP;
+    } else if (domain == "myqcloud" && tld == "com" && previous == "cos" && 
!last.empty()) {
+        return Vault::COS;
+    } else if (domain == "myhuaweicloud" && tld == "com" && previous == "obs" 
&& !last.empty()) {
+        return Vault::OBS;
+    } else if (domain == "bcebos" && tld == "com" && !last.empty()) {
+        return Vault::BOS;
+    } else if ((domain == "volces" || domain == "ivolces") && tld == "com" &&
+               ((last.starts_with("tos-") && is_region(last.substr(4))) ||
+                (last.starts_with("tos-s3-") && is_region(last.substr(7))))) {
+        return Vault::TOS;
+    } else if (last == "core" && (previous == "blob" || previous == "dfs") && 
!labels.empty() &&
+               ((domain == "windows" && tld == "net") ||
+                (domain == "chinacloudapi" && tld == "cn") ||
+                (domain == "usgovcloudapi" && tld == "net") ||
+                (domain == "cloudapi" && tld == "de"))) {
+        return Vault::AZURE;
+    }
+    return Vault::S3;
+}
+
+AwsCredentialImplementation aws_credential_implementation(std::string_view 
version) {
+    return version == "v2" ? AwsCredentialImplementation::V2 : 
AwsCredentialImplementation::V1;
+}
+
+void normalize_obj_storage_endpoint(ObjStorageClientConfig& client, 
std::string_view scheme) {
+    auto& endpoint = obj_storage_connection(client).endpoint;
+    if (auto* aws = std::get_if<AwsObjStorageConfig>(&client); aws && 
aws->use_sdk_endpoint) {
+        endpoint.clear();
+        aws->sdk_endpoint_http = scheme == "http";
+        return;
+    }
+    if (!endpoint.empty() && endpoint.find("://") == std::string::npos) {
+        endpoint.insert(0,
+                        scheme == "http" && 
!std::holds_alternative<AzureObjStorageConfig>(client)
+                                ? "http://";
+                                : "https://";);
+    }
+}
+
+namespace {
+using Props = ObjStorageProperties;
+using Error = std::optional<std::string>;
+template <class T>
+using Parsed = ObjStorageConfigParseResult<T>;
+
+std::string value(const Props& properties, const char* key) {
+    auto it = properties.find(key);
+    return it == properties.end() ? std::string {} : it->second;
+}
+
+template <class T>
+Error read_number(const Props& props, const char* key, std::optional<T>& 
output) {
+    auto it = props.find(key);
+    if (it == props.end()) {
+        return {};
+    }
+    const auto& text = it->second;
+    int number;
+    auto [end, error] = std::from_chars(text.data(), text.data() + 
text.size(), number);
+    if (error != std::errc {} || end != text.data() + text.size() || number < 
-1) {
+        return fmt::format("invalid {} value \"{}\"", key, text);
+    }
+    // Legacy resources use -1 or 0 for host defaults. Passing a zero request 
timeout
+    // to the SDK would disable stalled-transfer detection instead of keeping 
the BE default.
+    if (number > 0) {
+        output = T(number);
+    }
+    return {};
+}
+Error read_timeouts(const Props& props, ObjStorageConnectionConfig& conn) {
+    if (auto error = read_number(props, "AWS_REQUEST_TIMEOUT_MS", 
conn.request_timeout)) {
+        return error;
+    }
+    return read_number(props, "AWS_CONNECTION_TIMEOUT_MS", 
conn.connect_timeout);
+}
+Error read_bool(const Props& props, const char* key, bool& output) {
+    auto it = props.find(key);
+    if (it == props.end()) {
+        return {};
+    }
+    auto text = it->second;
+    std::transform(text.begin(), text.end(), text.begin(),
+                   [](unsigned char c) { return 
static_cast<char>(std::tolower(c)); });
+    if (text == "true") {
+        output = true;
+    } else if (text == "false") {
+        output = false;
+    } else {
+        return fmt::format("invalid {} value \"{}\"", key, it->second);
+    }
+    return {};
+}
+template <class C>
+Error read_regional_options(const Props& props, C& client) {
+    client.region = value(props, "AWS_REGION");
+    client.connection.endpoint = value(props, "AWS_ENDPOINT");
+    if (auto error = read_bool(props, "use_path_style", 
client.use_path_style)) {
+        return error;
+    }
+    bool override_endpoint = true;
+    if (auto error = read_bool(props, "AWS_NEED_OVERRIDE_ENDPOINT", 
override_endpoint)) {
+        return error;
+    }
+    if (!override_endpoint) {
+        if constexpr (std::is_same_v<C, AwsObjStorageConfig>) {
+            client.use_sdk_endpoint = true;
+        } else {
+            return "AWS SDK automatic endpoint selection requires 
provider=AWS";
+        }
+    }
+    if (auto error = read_number(props, "AWS_MAX_CONNECTIONS", 
client.connection.max_connections)) {
+        return error;
+    }
+    return read_timeouts(props, client.connection);
+}
+
+AwsCredentialChainType chain_source(CredProviderType mode) {
+    switch (mode) {
+    case CredProviderType::Env:
+        return AwsCredentialChainType::Environment;
+    case CredProviderType::SystemProperties:
+        return AwsCredentialChainType::Profile;
+    case CredProviderType::WebIdentity:
+        return AwsCredentialChainType::WebIdentity;
+    case CredProviderType::Container:
+        return AwsCredentialChainType::Container;
+    case CredProviderType::InstanceProfile:
+        return AwsCredentialChainType::InstanceProfile;
+    default:
+        return AwsCredentialChainType::Default;
+    }
+}
+
+Parsed<AccessKeyOrAnonymousObjStorageAuth> parse_access_key_credentials(const 
Props& props) {
+    AccessKeyObjStorageAuth keys {value(props, "AWS_ACCESS_KEY"), value(props, 
"AWS_SECRET_KEY"),
+                                  value(props, "AWS_TOKEN")};
+    if (!keys.access_key.empty() && !keys.secret_key.empty()) {
+        // Some persisted resources also carry AWS role/provider fields. 
Static keys
+        // have always taken precedence; they include temporary credentials 
issued by FE.
+        return AccessKeyOrAnonymousObjStorageAuth {std::move(keys)};
+    }
+    if (keys.access_key.empty() != keys.secret_key.empty()) {
+        return std::string("Incomplete access key credentials");
+    }
+    if (!keys.session_token.empty()) {
+        return std::string("Session token requires access key credentials");
+    }
+    const auto mode = to_upper(value(props, "AWS_CREDENTIALS_PROVIDER_TYPE"));
+    if ((!mode.empty() && mode != "DEFAULT" && mode != "SIMPLE" && mode != 
"ANONYMOUS") ||
+        !value(props, "AWS_ROLE_ARN").empty() || !value(props, 
"AWS_EXTERNAL_ID").empty()) {
+        return std::string(
+                "AWS credential chains and AssumeRole are not supported for 
this provider");
+    }
+    return AccessKeyOrAnonymousObjStorageAuth {AnonymousObjStorageAuth {}};

Review Comment:
   [P1] Preserve the GCS DEFAULT HMAC credential chain
   
   With `provider=GCP`, `AWS_CREDENTIALS_PROVIDER_TYPE=DEFAULT` and no explicit 
AK/SK or native `gs` credential, FE intentionally requests ambient HMAC 
credentials: `GcsAuthResolver` and `AbstractDelegatingS3Properties` explicitly 
preserve this legacy mode, and 
`S3ResourceTest.testLegacyGcpDefaultChainSurvivesReplayAndPolicyPush` covers 
its producer. The default v2 BE previously built 
`CustomAwsCredentialsProviderChain` and could read HMAC keys from 
environment/profile. The GCP parser now reaches this unconditional anonymous 
result, and `GcpObjStorageClient` constructs `AnonymousAWSCredentialsProvider`, 
so existing private GCS reads/loads become unsigned and fail after upgrade. 
Keep `DEFAULT` distinct from `ANONYMOUS` for this supported mode.



##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -1792,22 +1793,26 @@ void update_s3_resource(const TStorageResource& param, 
io::RemoteFileSystemSPtr
     Status st;
     io::RemoteFileSystemSPtr fs;
 
-    if (!existed_fs) {
-        // No such FS instance on BE
-        auto res = 
io::S3FileSystem::create(S3Conf::get_s3_conf(param.s3_storage_param),
-                                            std::to_string(param.id));
+    auto parsed = parse_be_obj_storage_config(param.s3_storage_param);
+    if (!parsed.has_value()) {
+        LOG(WARNING) << "invalid S3 storage resource " << param.id << ": " << 
parsed.error();
+        return;
+    }
+    auto storage_config = std::move(parsed).value();
+    auto current = 
std::static_pointer_cast<io::ObjStorageFileSystem>(existed_fs);
+    DCHECK(!existed_fs || existed_fs->type() == 
io::FileSystemType::OBJECT_STORAGE);
+    // A changed target needs its own filesystem. Open readers and writers 
still own the old holder.
+    if (current && current->same_target(storage_config)) {

Review Comment:
   [P2] Keep existing holders connected after provider changes
   
   A provider-only `ALTER RESOURCE` from S3 to GCP can keep the same 
`storage.googleapis.com` endpoint, `us-east1` region, bucket, prefix and HMAC 
keys; FE permits this even for policy resources. `same_client_target` rejects 
the variant change, so this code replaces the filesystem. 
`RowsetMeta::remote_storage_resource` caches its filesystem after the first 
lookup, and open readers retain that filesystem's holder. A later AK/SK 
rotation updates only the replacement filesystem in the registry, leaving old 
rowsets using revoked keys until reload/restart. Preserve or redirect the 
existing shared holder for same-physical-target provider changes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to