morningman opened a new pull request, #70725:
URL: https://github.com/apache/doris/pull/70725
### Background
Lance catalogs expose a namespace as Doris databases and tables. The FE
reads dataset manifests through the Lance Java SDK and passes the pinned
version and storage options to the BE's lance-c reader. Master now serves
filesystem namespaces through the Lance connector plugin; released 4.1/4.2
versions also serve REST namespaces and OSS storage.
### What problem does this PR solve?
Issue Number: #70680
Related PR: #70681 (filesystem plugin foundation), #66581 (REST namespace
source), #66805 (vended storage options source), #67157 (OSS source), #70338
(property masking), #70364 (endpoint checks with default ports)
PR Type: Feature
Problem Summary:
This ports REST namespaces, namespace-vended credentials and OSS support
from branch-4.x to the master Lance plugin. Users can read tables through a
namespace service without configuring static dataset credentials in Doris. The
change also validates REST authentication and headers, declares the new
endpoint and secrets, and adds the REST fixture and behavior tests. Filesystem
catalogs keep their existing defaults and checks.
Original authors: @zhangstar333 and @FANNG1.
### Details
- **Change map:** Production code +465/-58 lines; tests and fixtures
+1351/-47; generated regression output +20; pipeline configuration +3. Most
changed lines are tests or fixture infrastructure.
| Part | Purpose | Approximate changed production lines |
| --- | --- | --- |
| REST namespace access | Validate properties, select the namespace
client, request table credentials, and sanitize errors | 290 |
| OSS storage | Adapt the engine's OSS binding and normalize provider
options; refuse OSS-HDFS and unresolved AssumeRole | 200 |
| Vended S3 identity | Validate after the merge and prevent an independent
vended key pair inheriting another identity's session token | 30 |
- **Scope:** REST authentication supports `none`, `bearer` and `api_key`,
plus extra headers; namespace-vended options override catalog options for FE
metadata and BE scans. OSS uses Lance's OpenDAL provider. Declared-only and
namespace-managed-version tables fail explicitly; their version protocol
belongs to a later phase.
- **Design:** Reuse the namespace and storage-provider contracts from
4.1/4.2 inside the existing plugin, without adding Lance branches to the
engine. The four-phase plan is tracked in #70680. Reading storage-latest for
managed tables was rejected because the namespace may not have published that
version; this phase refuses them instead. Credential validation follows the
catalog/vended merge so an endpoint-only catalog can receive valid credentials.
- **Usage:** With the external regression namespace fixture:
```sql
CREATE CATALOG lance_rest PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "rest",
"lance.rest.uri" = "http://127.0.0.1:19102",
"lance.rest.security.type" = "bearer",
"lance.rest.bearer-token" = "doris-lance-rest-test-token",
"s3.endpoint" = "http://127.0.0.1:19001",
"s3.region" = "us-east-1",
"use_path_style" = "true"
);
SELECT count(*), sum(row_id) FROM lance_rest.`default`.all_types;
```
The committed fixture returns `12, 78` without a static access key or
secret key in the catalog.
- **Default behavior:** `lance.catalog.type` remains `filesystem`; REST
authentication defaults to `none`. REST requires `lance.rest.uri` and rejects
`warehouse`. No new FE configuration or session variable is introduced.
- **Compatibility:** Preserve the 4.1/4.2 property names and raw catalog
map. Catalog reconstruction and replay perform no native connection; first use
opens the namespace. No shared SPI, Thrift, Protobuf or storage-format change;
the plugin API stays at master's 34.0. Reads retain the foundation's FE/BE
compatibility checks. A downgrade needs a version that already supports the
chosen Lance namespace and storage provider.
- **Security and privileges:** `lance.rest.uri` is a user-controlled
outbound HTTP/SSRF surface. Catalog creation requires scoped `CREATE_PRIV`,
endpoint changes require `ALTER_PRIV`, and reads require `SELECT_PRIV`. Declare
REST alongside all existing S3-compatible endpoint aliases so the configured
engine gate covers CREATE and ALTER. Mask bearer/API-key values and
credential-bearing extra headers; strip them from provider errors. Existing
engine audit handling is reused.
- **Follow-up plan:** Predicate/LIMIT/COUNT pushdown and caches remain later
phases of #70680; namespace-managed versioning remains a separate follow-up.
### Testing
- Locally, 118 Lance unit tests and 13 SPI security/surface tests passed,
with zero failures or skips. The Lance tests include real JNI, REST
authentication/failures, provider options, and native subprocess checks with
unrelated AWS credentials and endpoints in the environment.
- `run-fe-ut.sh` passed all 118 Lance tests plus 77 engine tests for catalog
privileges/endpoint gating, plugin registration and 4.1.5 image replay.
- Eleven targeted mutations failed on their intended cases after a passing
62-test control, covering credential merge/identity, vending, managed tables,
OSS routing, endpoint declarations, masking and error sanitizing.
- The new missing-session-token test failed before the identity fix and
passed afterward; a complete vended key pair must not inherit a catalog or
ambient session token.
- `test_lance_catalog_all_types` and `test_lance_rest_catalog` passed
together on freshly built FE/ASAN BE. The REST output was generated with the
regression runner; all four shared result blocks match the independent
branch-4.2 output. The local S3 service validates SigV4 and serves the exact
committed dataset; the shipped REST stub ran on the host.
- A non-SUPER user was denied CREATE/ALTER without scoped grants, succeeded
after grants, and read the REST schema and all 12 rows with SELECT permission.
SHOW CREATE masked the bearer token and credential-bearing extra header.
### Notes for Reviewers
- Keep the filesystem AssumeRole rejection and all endpoint aliases added
during the foundation review. Only the obsolete REST-rejection regression
assertion is removed.
- A complete namespace-vended S3 key pair without a token explicitly sets an
empty token before merging. This blocks a catalog STS token and native
environment fallback; an explicitly vended token is preserved. The same issue
is recorded for the 4.x maintainers.
- The Docker daemon cannot read cached image blobs on this host. Full
MinIO/Compose, remote OSS and Linux execution are not claimed; the External
pipeline enables the new suite and its Compose fixture.
### Release note
Lance catalogs support REST namespace services, namespace-vended credentials
and OSS storage through the connector plugin. Existing 4.1/4.2 catalog property
names remain supported; namespace-managed-version tables report an explicit
unsupported error until that protocol is added.
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: Yes (extend the Lance catalog documentation
in apache/doris-website with REST authentication, OSS, vended credentials and
the managed-versioning limitation alongside the plugin port)
- Pre-submission status: Partially verified: full MinIO/Compose, remote OSS,
Linux and the complete FE unit suite were not run locally
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5) for the original port; Codex (GPT-6)
for the current-master adaptation
- Generated: code, tests, REST fixture, validation helpers and this
description
- Human review: not stated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]