xylaaaaa opened a new pull request, #66782:
URL: https://github.com/apache/doris/pull/66782
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Databricks Unity Catalog returns short-lived Azure SAS credentials in
Iceberg REST `loadTable` responses using keys such as
`adls.sas-token.<account-host>`. Doris previously dropped `adls.*` properties
during credential filtering, and the Iceberg connector package did not include
`ADLSFileIO`. Therefore catalog and table discovery could succeed while reading
table data from `abfss://` failed.
This change:
- preserves and converts Iceberg ADLS SAS properties to the account-scoped
Hadoop ABFS configuration consumed by Doris BE;
- packages Iceberg Azure FileIO and its shaded Azure SDK runtime in the
Iceberg connector;
- prevents short-lived storage credentials from being exposed by the
existing HDFS configuration INFO log;
- adds tests for the Databricks/Iceberg ADLS credential shape, Hadoop ABFS
consumption, and `abfss://` FileIO resolution.
The implementation consumes the SAS token returned for the current table
load. Refreshing credentials during a long-running query remains separate
follow-up work. A live customer-workspace retry is also still required for
end-to-end confirmation.
### Release note
Support Azure SAS credentials vended by Iceberg REST catalogs such as
Databricks Unity Catalog.
### Check List (For Author)
- Test:
- Unit Test: `CredentialUtilsTest`, `DefaultConnectorContextVendTest`,
and `IcebergScanPlanProviderTest` (162 tests passed)
- Manual test: `./build.sh --fe`
- Manual test: `./build.sh --be -j 8`
- Manual test: `build-support/check-format.sh`
- Manual test: `git diff --check`
- Behavior changed: Yes. Iceberg REST catalogs can pass vended Azure SAS
credentials through to ADLS reads.
- Does this need documentation: No
`clang-tidy` could not analyze `hdfs_builder.cpp` in this workspace because
the Hadoop HDFS include transitively failed to find `stdarg.h`; the complete BE
build, including this file and final `doris_be` link, passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
