sollhui opened a new pull request, #70753: URL: https://github.com/apache/doris/pull/70753
### Background Iceberg REST catalogs can return temporary storage credentials with table metadata. Doris must use those credentials for FE metadata access and BE data access, while the REST catalog owns the authentication needed to renew them. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Catalog-issued `gcs.oauth2.token` credentials are filtered out or cannot be represented by the native GCS authentication path. Queries can consequently use the wrong storage identity or fail authentication. Support the supplied token, refresh it on FE using live catalog authentication, and send a storage-only snapshot to BE. The change spans the refactored filesystem, Iceberg and object-storage modules because credential selection, renewal, serialization and client lifetime must agree. ### Details - **Change map:** Production +923/-57 lines, tests +805/-1, build +47/-0; no generated files or documentation changes. About 435 production lines implement catalog-owned renewal and shared token state; the remaining changes carry tokens through FE/BE authentication, isolate BE snapshots, propagate normalization failures and release token-specific clients. - **Frequency:** Always for the reproduced token-only property binding; live Polaris/GCS queries were not exercised. - **Root cause:** Cloud-property filtering omits `gcs.*`, native GCS credentials lack an access-token mode, and storage normalization treats conversion errors as absent credentials. - **Why existing tests missed it:** Existing GCS authentication tests cover ADC, VM identity, impersonation and HMAC rather than catalog tokens. The closest external vended-credential regression targets S3/OSS/COS and is disabled for unavailable infrastructure; it does not exercise GCS renewal. - **Fix:** Recognize token aliases, include token/expiry in cache identity, retain per-table and delegated-user authentication during refresh, and close refresh state with the catalog. BE uses weak, bucket-isolated client caching for supplied tokens. Temporary tokens cannot become persisted resource/vault credentials. - **Other paths and branches:** Scan and write planning share credential extraction. JNI system-table serialization strips catalog/user secrets and FE context while preserving a current storage token. Other authentication modes keep their existing cache behavior. Release branches and the introducing commit were not established; this targets master only. - **Behavior change:** Invalid vended credentials now fail instead of falling back to static catalog credentials, including normalization failures for other providers. No new configuration is required. BE does not renew a running query's token; expiry can still fail that query. ### Testing - Locally passed 69 tests in the GCS and shared Iceberg modules: 13 resolver, 8 authentication, 26 filesystem-property and 22 refresh/lifecycle cases, using Iceberg 1.12.0 and a local HTTP service. - An isolated SDK harness passed the 22 refresh cases plus the actual system-table FileIO serialization test. The latter checks that only the storage token survives transport. - The same minimal resolver assertion fails on unmodified master and passes on this change. Production C++ credential parsing compiled with C++20 and passed 26 boundary cases. - Added BE weak-cache/parser tests and FE bridge tests; these have not run. Full FE/BE builds, full connector tests, mixed-version execution and live GCS integration remain unverified because generated sources and third-party build dependencies are unavailable locally. ### Notes for Reviewers Renewal performs outbound FE HTTP requests to catalog-configured or server-vended credential endpoints. Setting a catalog endpoint requires CREATE/ALTER CATALOG privileges; the user-configured refresh endpoint is registered with the existing conditional SSRF gate. Review the REST session adapter and credential snapshot boundary first. FE filesystem code remains independent of Iceberg. ### Release note Support Iceberg REST catalog-issued GCS OAuth2 tokens and FE renewal for subsequent queries. Invalid vended credentials now report errors instead of silently falling back to another storage identity. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; document GCS vended-credential support, renewal and running-query expiry limitations in the Iceberg REST catalog guide. - Pre-submission status: Partially verified: full FE/BE and connector integration tests, mixed-version operation and live GCS integration remain unverified. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code adaptation, tests and this description. - Human review: not stated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
