xylaaaaa opened a new pull request, #70739:
URL: https://github.com/apache/doris/pull/70739

   ### Background
   
   Doris plans external Azure Blob access in FE and opens the data files in BE. 
#70657 added SharedKey, SAS and Entra client-secret authentication to the 
shared object-storage client, together with a reusable Thrift credential type. 
File scan and sink requests still carry the older storage property map.
   
   ### What problem does this PR solve?
   
   Issue Number: #68103 (tracking)
   
   Related PR: #70657 (shared Azure client and credential type)
   
   PR Type: Feature
   
   Problem Summary: When an Iceberg REST catalog vends a table-scoped Azure 
SAS, the native reader and writer need the same credential that FE selected for 
that table. The same gap affects a native file operation configured with Entra 
client credentials. This change carries Azure access through schema inference, 
scans and writes, and binds it to the file's storage account and container when 
BE opens the file.
   
   ### Details
   
   - **Change map:** FE production code and plugin dependencies (+667/-77, 21 
files) translate and bind Azure credentials. BE production code (+405/-70, 27 
files) carries them to file readers and writers. Two Thrift files (+10) add 
eight optional fields. Ten unit-test files (+1,011/-3) exercise these 
boundaries.
   - **Scope:** Native `FILE_S3` access to Azure in external scans, schema and 
Parquet metadata reads, Hive and Iceberg sinks, TVF inserts, and 
`OUTFILE`/`EXPORT`. Backup repositories and Broker Load retain their supported 
auth paths and report a planning error for unsupported Azure token credentials.
   - **Design:** FE translates Iceberg `adls.sas-token.<host>` and its optional 
expiry into storage properties for that table. A single adapter converts 
provider properties to `TAzureClientConfig`. BE resolves each file's container 
and key, checks explicitly configured account/container bindings, and passes 
the resulting resource to the shared client. A request carries credentials for 
one Azure storage account.
   - **Usage:** Set `provider=azure` and `azure.auth_type=SAS` with the Blob 
endpoint and SAS token, or select `OAuth2` with the Entra client-credential 
properties. An Iceberg REST catalog can supply the SAS per table.
   - **Default behavior:** SharedKey remains the default Azure auth type. 
Requests without a typed Azure field use the existing property-map path.
   - **Compatibility:** The additions are optional. New BEs accept requests 
from older FEs. Upgrade participating BEs before enabling native Azure 
credentials and locations from the new FE.
   - **Behavior change:** A vended Azure SAS can authorize a native table 
operation. Conflicting Azure SAS tokens or incompatible account/container 
locations surface an error before the client opens the wrong target.
   
   ### Testing
   
   - On the rebased branch, ran eight focused FE suites covering Azure 
properties, Thrift binding, catalog vending, scan requests, URI routing and 
EXPORT. **66 passed, 0 failed, 1 existing test skipped.** FE Checkstyle, 
clang-format 16, changed-line clang-tidy and BE header hygiene passed.
   - Checked all eight Thrift field IDs against master and branches 4.2, 4.1, 
4.0 and 3.1. Encoded a table-vended SAS using the rebased Java code and decoded 
both the scan field (43) and the OUTFILE field (24) with the regenerated C++ 
types, including the Blob endpoint and expiry.
   - Before rebasing, the integration build passed end-to-end tests with a full 
FE and an Azure-enabled ASAN BE. Doris read a SAS-protected Parquet on Azure. 
Against real Azure Databricks Unity Catalog, it inserted into Iceberg with a 
REST-vended SAS and read the same three rows after reloading the table. On 
Azure Blob, Entra client credentials and SharedKey each passed `SELECT INTO 
OUTFILE` followed by an exact-file `S3()` read. Independent Azure SDK reads 
matched the Doris results.
   
   ### Notes for Reviewers
   
   The configurable Blob endpoint and Entra authority are outgoing HTTP(S) 
destinations for FE and BE and therefore an SSRF surface. A user with 
permission to create or alter an external catalog, or to execute an 
external-file TVF with inline storage properties, can select these 
destinations. Using a named resource in the TVF also requires USAGE on that 
resource. An Iceberg REST catalog can supply the storage account host with a 
vended SAS.
   
   ### Release note
   
   Native Azure external file reads and writes can use SAS or Microsoft Entra 
client credentials. Iceberg REST catalogs can vend a table-scoped Azure SAS for 
native data access.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: Yes. Document the auth properties, Iceberg 
REST credential vending and BE-first rollout in the Azure storage and Iceberg 
catalog guides.
   - Pre-submission status: Partially verified. The rebased local ASAN BE 
unit-test build is in progress.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex (GPT-6).
   - Generated: Codex assisted with the integration patch, unit tests, rebase, 
validation and this description.
   - Human review: The contributor confirmed the integration scope and the live 
test results. Human review of the rebased code is pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to