morningman opened a new pull request, #70725:
URL: https://github.com/apache/doris/pull/70725

   ### Background
   
   Lance catalogs expose a namespace as Doris databases and tables. The FE 
reads dataset manifests through the Lance Java SDK and passes the pinned 
version and storage options to the BE's lance-c reader. Master now serves 
filesystem namespaces through the Lance connector plugin; released 4.1/4.2 
versions also serve REST namespaces and OSS storage.
   
   ### What problem does this PR solve?
   
   Issue Number: #70680
   Related PR: #70681 (filesystem plugin foundation), #66581 (REST namespace 
source), #66805 (vended storage options source), #67157 (OSS source), #70338 
(property masking), #70364 (endpoint checks with default ports)
   PR Type: Feature
   
   Problem Summary:
   
   This ports REST namespaces, namespace-vended credentials and OSS support 
from branch-4.x to the master Lance plugin. Users can read tables through a 
namespace service without configuring static dataset credentials in Doris. The 
change also validates REST authentication and headers, declares the new 
endpoint and secrets, and adds the REST fixture and behavior tests. Filesystem 
catalogs keep their existing defaults and checks.
   
   Original authors: @zhangstar333 and @FANNG1.
   
   ### Details
   
   - **Change map:** Production code +465/-58 lines; tests and fixtures 
+1351/-47; generated regression output +20; pipeline configuration +3. Most 
changed lines are tests or fixture infrastructure.
   
     | Part | Purpose | Approximate changed production lines |
     | --- | --- | --- |
     | REST namespace access | Validate properties, select the namespace 
client, request table credentials, and sanitize errors | 290 |
     | OSS storage | Adapt the engine's OSS binding and normalize provider 
options; refuse OSS-HDFS and unresolved AssumeRole | 200 |
     | Vended S3 identity | Validate after the merge and prevent an independent 
vended key pair inheriting another identity's session token | 30 |
   
   - **Scope:** REST authentication supports `none`, `bearer` and `api_key`, 
plus extra headers; namespace-vended options override catalog options for FE 
metadata and BE scans. OSS uses Lance's OpenDAL provider. Declared-only and 
namespace-managed-version tables fail explicitly; their version protocol 
belongs to a later phase.
   - **Design:** Reuse the namespace and storage-provider contracts from 
4.1/4.2 inside the existing plugin, without adding Lance branches to the 
engine. The four-phase plan is tracked in #70680. Reading storage-latest for 
managed tables was rejected because the namespace may not have published that 
version; this phase refuses them instead. Credential validation follows the 
catalog/vended merge so an endpoint-only catalog can receive valid credentials.
   - **Usage:** With the external regression namespace fixture:
   
     ```sql
     CREATE CATALOG lance_rest PROPERTIES (
         "type" = "lance",
         "lance.catalog.type" = "rest",
         "lance.rest.uri" = "http://127.0.0.1:19102";,
         "lance.rest.security.type" = "bearer",
         "lance.rest.bearer-token" = "doris-lance-rest-test-token",
         "s3.endpoint" = "http://127.0.0.1:19001";,
         "s3.region" = "us-east-1",
         "use_path_style" = "true"
     );
     SELECT count(*), sum(row_id) FROM lance_rest.`default`.all_types;
     ```
   
     The committed fixture returns `12, 78` without a static access key or 
secret key in the catalog.
   - **Default behavior:** `lance.catalog.type` remains `filesystem`; REST 
authentication defaults to `none`. REST requires `lance.rest.uri` and rejects 
`warehouse`. No new FE configuration or session variable is introduced.
   - **Compatibility:** Preserve the 4.1/4.2 property names and raw catalog 
map. Catalog reconstruction and replay perform no native connection; first use 
opens the namespace. No shared SPI, Thrift, Protobuf or storage-format change; 
the plugin API stays at master's 34.0. Reads retain the foundation's FE/BE 
compatibility checks. A downgrade needs a version that already supports the 
chosen Lance namespace and storage provider.
   - **Security and privileges:** `lance.rest.uri` is a user-controlled 
outbound HTTP/SSRF surface. Catalog creation requires scoped `CREATE_PRIV`, 
endpoint changes require `ALTER_PRIV`, and reads require `SELECT_PRIV`. Declare 
REST alongside all existing S3-compatible endpoint aliases so the configured 
engine gate covers CREATE and ALTER. Mask bearer/API-key values and 
credential-bearing extra headers; strip them from provider errors. Existing 
engine audit handling is reused.
   - **Follow-up plan:** Predicate/LIMIT/COUNT pushdown and caches remain later 
phases of #70680; namespace-managed versioning remains a separate follow-up.
   
   ### Testing
   
   - Locally, 118 Lance unit tests and 13 SPI security/surface tests passed, 
with zero failures or skips. The Lance tests include real JNI, REST 
authentication/failures, provider options, and native subprocess checks with 
unrelated AWS credentials and endpoints in the environment.
   - `run-fe-ut.sh` passed all 118 Lance tests plus 77 engine tests for catalog 
privileges/endpoint gating, plugin registration and 4.1.5 image replay.
   - Eleven targeted mutations failed on their intended cases after a passing 
62-test control, covering credential merge/identity, vending, managed tables, 
OSS routing, endpoint declarations, masking and error sanitizing.
   - The new missing-session-token test failed before the identity fix and 
passed afterward; a complete vended key pair must not inherit a catalog or 
ambient session token.
   - `test_lance_catalog_all_types` and `test_lance_rest_catalog` passed 
together on freshly built FE/ASAN BE. The REST output was generated with the 
regression runner; all four shared result blocks match the independent 
branch-4.2 output. The local S3 service validates SigV4 and serves the exact 
committed dataset; the shipped REST stub ran on the host.
   - A non-SUPER user was denied CREATE/ALTER without scoped grants, succeeded 
after grants, and read the REST schema and all 12 rows with SELECT permission. 
SHOW CREATE masked the bearer token and credential-bearing extra header.
   
   ### Notes for Reviewers
   
   - Keep the filesystem AssumeRole rejection and all endpoint aliases added 
during the foundation review. Only the obsolete REST-rejection regression 
assertion is removed.
   - A complete namespace-vended S3 key pair without a token explicitly sets an 
empty token before merging. This blocks a catalog STS token and native 
environment fallback; an explicitly vended token is preserved. The same issue 
is recorded for the 4.x maintainers.
   - The Docker daemon cannot read cached image blobs on this host. Full 
MinIO/Compose, remote OSS and Linux execution are not claimed; the External 
pipeline enables the new suite and its Compose fixture.
   
   ### Release note
   
   Lance catalogs support REST namespace services, namespace-vended credentials 
and OSS storage through the connector plugin. Existing 4.1/4.2 catalog property 
names remain supported; namespace-managed-version tables report an explicit 
unsupported error until that protocol is added.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: Yes (extend the Lance catalog documentation 
in apache/doris-website with REST authentication, OSS, vended credentials and 
the managed-versioning limitation alongside the plugin port)
   - Pre-submission status: Partially verified: full MinIO/Compose, remote OSS, 
Linux and the complete FE unit suite were not run locally
   
   ### AI Disclosure
   
   - Tool: Claude Code (Claude Opus 5.5) for the original port; Codex (GPT-6) 
for the current-master adaptation
   - Generated: code, tests, REST fixture, validation helpers and this 
description
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to