xylaaaaa opened a new pull request, #70739: URL: https://github.com/apache/doris/pull/70739
### Background Doris plans external Azure Blob access in FE and opens the data files in BE. #70657 added SharedKey, SAS and Entra client-secret authentication to the shared object-storage client, together with a reusable Thrift credential type. File scan and sink requests still carry the older storage property map. ### What problem does this PR solve? Issue Number: #68103 (tracking) Related PR: #70657 (shared Azure client and credential type) PR Type: Feature Problem Summary: When an Iceberg REST catalog vends a table-scoped Azure SAS, the native reader and writer need the same credential that FE selected for that table. The same gap affects a native file operation configured with Entra client credentials. This change carries Azure access through schema inference, scans and writes, and binds it to the file's storage account and container when BE opens the file. ### Details - **Change map:** FE production code and plugin dependencies (+667/-77, 21 files) translate and bind Azure credentials. BE production code (+405/-70, 27 files) carries them to file readers and writers. Two Thrift files (+10) add eight optional fields. Ten unit-test files (+1,011/-3) exercise these boundaries. - **Scope:** Native `FILE_S3` access to Azure in external scans, schema and Parquet metadata reads, Hive and Iceberg sinks, TVF inserts, and `OUTFILE`/`EXPORT`. Backup repositories and Broker Load retain their supported auth paths and report a planning error for unsupported Azure token credentials. - **Design:** FE translates Iceberg `adls.sas-token.<host>` and its optional expiry into storage properties for that table. A single adapter converts provider properties to `TAzureClientConfig`. BE resolves each file's container and key, checks explicitly configured account/container bindings, and passes the resulting resource to the shared client. A request carries credentials for one Azure storage account. - **Usage:** Set `provider=azure` and `azure.auth_type=SAS` with the Blob endpoint and SAS token, or select `OAuth2` with the Entra client-credential properties. An Iceberg REST catalog can supply the SAS per table. - **Default behavior:** SharedKey remains the default Azure auth type. Requests without a typed Azure field use the existing property-map path. - **Compatibility:** The additions are optional. New BEs accept requests from older FEs. Upgrade participating BEs before enabling native Azure credentials and locations from the new FE. - **Behavior change:** A vended Azure SAS can authorize a native table operation. Conflicting Azure SAS tokens or incompatible account/container locations surface an error before the client opens the wrong target. ### Testing - On the rebased branch, ran eight focused FE suites covering Azure properties, Thrift binding, catalog vending, scan requests, URI routing and EXPORT. **66 passed, 0 failed, 1 existing test skipped.** FE Checkstyle, clang-format 16, changed-line clang-tidy and BE header hygiene passed. - Checked all eight Thrift field IDs against master and branches 4.2, 4.1, 4.0 and 3.1. Encoded a table-vended SAS using the rebased Java code and decoded both the scan field (43) and the OUTFILE field (24) with the regenerated C++ types, including the Blob endpoint and expiry. - Before rebasing, the integration build passed end-to-end tests with a full FE and an Azure-enabled ASAN BE. Doris read a SAS-protected Parquet on Azure. Against real Azure Databricks Unity Catalog, it inserted into Iceberg with a REST-vended SAS and read the same three rows after reloading the table. On Azure Blob, Entra client credentials and SharedKey each passed `SELECT INTO OUTFILE` followed by an exact-file `S3()` read. Independent Azure SDK reads matched the Doris results. ### Notes for Reviewers The configurable Blob endpoint and Entra authority are outgoing HTTP(S) destinations for FE and BE and therefore an SSRF surface. A user with permission to create or alter an external catalog, or to execute an external-file TVF with inline storage properties, can select these destinations. Using a named resource in the TVF also requires USAGE on that resource. An Iceberg REST catalog can supply the storage account host with a vended SAS. ### Release note Native Azure external file reads and writes can use SAS or Microsoft Entra client credentials. Iceberg REST catalogs can vend a table-scoped Azure SAS for native data access. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes. Document the auth properties, Iceberg REST credential vending and BE-first rollout in the Azure storage and Iceberg catalog guides. - Pre-submission status: Partially verified. The rebased local ASAN BE unit-test build is in progress. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Codex assisted with the integration patch, unit tests, rebase, validation and this description. - Human review: The contributor confirmed the integration scope and the live test results. Human review of the rebased code is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
