FANNG1 opened a new issue, #66772: URL: https://github.com/apache/doris/issues/66772
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version `branch-4.1` @ `0e53b31f586`. The FE Lance catalog currently exists only on `branch-4.1` — `master` has no `fe/fe-core/src/main/java/org/apache/doris/datasource/lance/`. `LanceStorageOptions` came in with e3289c1a5df (#65730); the REST properties were reshaped by 8375559ce88 (#66581). Server side: **Apache Gravitino 1.3.0**, native `lance-rest` auxiliary service (`gravitino.auxService.names = lance-rest`, `gravitino.lance-rest.httpPort = 9101`, `namespace-backend = gravitino`), base path `/lance`. Namespace library versions differ across the wire but are not the cause of either problem below: Gravitino bundles `lance-namespace-core` 0.4.5, while Doris FE pulls `lance-namespace-apache-client` 0.7.7 transitively from `org.lance:lance-core:9.1.0-beta.3`. ### What's Wrong? I ran the Lance REST catalog against a real Lance Namespace server (Gravitino) rather than the Python stub in `docker/thirdparties/docker-compose/iceberg/scripts/lance_rest_server.py`, and hit two independent problems. The protocol itself lines up — endpoint paths, the `$` delimiter, pagination, `managed_versioning`, and the `table_uri`/`location` fallback all match, and once both problems are worked around **every read path passes** (see *Anything else?*). These are integration defects, not a protocol mismatch. --- #### Problem 1 — namespace-vended storage credentials are silently dropped (blocking) Credential vending is the main reason to prefer a REST catalog over a filesystem one, and Doris explicitly asks for it — `LanceExternalCatalog.describeTable` sends `.vendCredentials(true)` for REST catalogs. Gravitino answers with **unprefixed** object-store option names (it stores them as `lance.storage.<key>` and strips that prefix in `LancePropertiesUtils.resolveLanceStorageOptions`): ```json "storage_options": { "access_key_id": "minioadmin", "secret_access_key": "minioadmin", "endpoint": "http://127.0.0.1:9000";, "region": "us-east-1", "allow_http": "true" } ``` But `LanceStorageOptions.forBackend` only ever looks up five `aws_`-prefixed names: ```java // fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageOptions.java S3_KEYS.put("AWS_ACCESS_KEY", "aws_access_key_id"); S3_KEYS.put("AWS_SECRET_KEY", "aws_secret_access_key"); S3_KEYS.put("AWS_TOKEN", "aws_session_token"); S3_KEYS.put("AWS_ENDPOINT", "aws_endpoint"); S3_KEYS.put("AWS_REGION", "aws_region"); ... S3_KEYS.forEach((dorisKey, lanceKey) -> putIfNotEmpty(result, dorisKey, lanceStorageOptions.get(lanceKey))); ``` Nothing matches, so **every vended credential is discarded before it reaches the BE**. The resulting failure is split across FE and BE, which makes it hard to read. The FE merges the vended map in raw and hands it to the Lance Java SDK, whose object-store layer accepts the unprefixed spelling — so all metadata operations succeed. Only the BE, which goes through `forBackend`, ends up with nothing: ``` DESC lance_novend.doris_probe.rest_probe; -- OK, 5 columns SELECT count(*) FROM lance_novend.doris_probe.rest_probe; ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR] open Lance dataset failed: LanceError(IO): Generic N/A error: ... Failed to get AWS credentials: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), .../lance-io/src/object_store/providers/aws.rs:401:21 ``` **Positive control isolating the key spelling.** I created a second Gravitino catalog whose properties spell the same values `lance.storage.aws_access_key_id`, `lance.storage.aws_secret_access_key`, `lance.storage.aws_endpoint`, `lance.storage.aws_region`, pointed it at the *same physical dataset*, and read it through the *same* Doris configuration (still no static credentials). The scan returned all 1024 rows. Key spelling was the only variable. Both spellings are valid for the Rust `object_store` config that ultimately consumes them, and the FE's success is direct evidence that the unprefixed form works — only Doris's own BE-side converter insists on one of the two. --- #### Problem 2 — the parent namespace is published as a database that cannot be listed `LanceExternalCatalog.listDatabaseNames` unconditionally publishes the configured root database: ```java // The configured root database represents the empty relative Lance namespace. LinkedHashSet<String> databases = new LinkedHashSet<>(); databases.add(rootDatabase); ``` so `default` always appears in `SHOW DATABASES`. It maps to the parent namespace itself, which is one level deep — but Gravitino's `ListTables` requires an identifier of exactly two levels (catalog + schema), so listing it fails: ``` SHOW TABLES FROM lance_gvt.`default`; ERROR 1105 (HY000): errCode = 2, detailMessage = Invalid input: Expected 2-level namespace but got: 1 ``` The same collision makes `"test_connection" = "true"` unusable, because the probe lists tables directly under the parent: ```java testNamespace.listTables(new ListTablesRequest().id(parent).limit(1)); testNamespace.listNamespaces(new ListNamespacesRequest().id(parent).limit(1)); ``` With `test_connection = true`, `CREATE CATALOG` fails outright with the same message. Leaving it at its default of `false` is the workaround. This is not a data-correctness problem — Doris loads databases lazily, so `default` is just an entry that must be routed around — but it does mean `SHOW DATABASES` advertises something unusable, and it silently rules out the connectivity check. ### What You Expected? 1. Vended `storage_options` should reach the BE regardless of which accepted alias the namespace server uses for a key. A server that vends `access_key_id` should work exactly like one that vends `aws_access_key_id` — as it already does on the FE side. 2. A database returned by `SHOW DATABASES` should be listable, and `test_connection = true` should not require the parent namespace itself to be table-listable. ### How to Reproduce? **1. Gravitino 1.3.0 with the Lance REST service**, and a `lakehouse-generic` catalog carrying unprefixed storage options: ```properties # conf/gravitino.conf gravitino.auxService.names = lance-rest gravitino.lance-rest.classpath = lance-rest-server/libs gravitino.lance-rest.httpPort = 9101 gravitino.lance-rest.namespace-backend = gravitino gravitino.lance-rest.gravitino-uri = http://127.0.0.1:8090 gravitino.lance-rest.gravitino-metalake = test ``` ```bash curl -X POST http://127.0.0.1:8090/api/metalakes/test/catalogs \ -H 'Accept: application/vnd.gravitino.v1+json' -H 'Content-Type: application/json' \ -d '{"name":"lance_catalog","type":"relational","provider":"lakehouse-generic","properties":{ "location":"s3://contacts/raw/lance", "lance.storage.access_key_id":"minioadmin", "lance.storage.secret_access_key":"minioadmin", "lance.storage.endpoint":"http://127.0.0.1:9000";, "lance.storage.region":"us-east-1", "lance.storage.allow_http":"true"}}' curl -X POST 'http://127.0.0.1:9101/lance/v1/namespace/lance_catalog%24doris_probe/create?delimiter=%24' \ -H 'Content-Type: application/json' \ -d '{"id":["lance_catalog","doris_probe"],"mode":"CREATE","properties":{}}' ``` **2. Write a dataset and register it** (pylance 7.0.0): ```python import lance, numpy as np, pyarrow as pa URI = "s3://contacts/raw/lance/doris_probe/rest_probe/" SO = {"access_key_id": "minioadmin", "secret_access_key": "minioadmin", "endpoint": "http://127.0.0.1:9000";, "region": "us-east-1", "allow_http": "true", "virtual_hosted_style_request": "false"} rid = np.arange(1, 1025, dtype=np.int64) emb = (rid[:, None] - 1).astype(np.float32) + np.arange(16, dtype=np.float32)[None, :] t = pa.table({ "row_id": pa.array(rid, type=pa.int64()), "int32_value": pa.array((rid % 100).astype(np.int32), type=pa.int32()), "embedding": pa.FixedSizeListArray.from_arrays( pa.array(emb.reshape(-1), type=pa.float32()), 16)}) lance.write_dataset(t, URI, mode="overwrite", storage_options=SO, max_rows_per_file=512) lance.dataset(URI, storage_options=SO).create_index( "embedding", index_type="IVF_FLAT", metric="l2", num_partitions=4) ``` ```bash curl -X POST 'http://127.0.0.1:9101/lance/v1/table/lance_catalog%24doris_probe%24rest_probe/register?delimiter=%24' \ -H 'Content-Type: application/json' \ -d '{"id":["lance_catalog","doris_probe","rest_probe"], "location":"s3://contacts/raw/lance/doris_probe/rest_probe/"}' ``` **3. Problem 1** — a catalog with no static credentials, relying entirely on vending: ```sql CREATE CATALOG lance_novend PROPERTIES ( "type" = "lance", "lance.catalog.type" = "rest", "lance.rest.uri" = "http://127.0.0.1:9101/lance";, "lance.namespace.parent" = "lance_catalog" ); DESC lance_novend.doris_probe.rest_probe; -- OK SELECT count(*) FROM lance_novend.doris_probe.rest_probe; -- CredentialsNotLoaded ``` **4. Problem 2** — against the same catalog: ```sql SHOW DATABASES FROM lance_novend; -- lists `default` SHOW TABLES FROM lance_novend.`default`; -- Expected 2-level namespace but got: 1 ``` and adding `"test_connection" = "true"` to the properties above makes `CREATE CATALOG` itself fail. ### Anything Else? **Everything else works.** With both problems worked around — static `s3.access_key`/`s3.secret_key`, and `test_connection` left at its default — the REST catalog reads Gravitino-managed Lance tables correctly. Verified against a deterministic fixture (1024 rows in 2 fragments, 16-dim float32 `embedding` where row *n* is `[n-1, n, …, n+14]`, so the exact squared L2 distance between rows *n* and *r* is `16·(n−r)²`, plus an IVF_FLAT index over 4 partitions), cross-checked row-for-row against pylance: | Layer | Result | |---|---| | `CREATE CATALOG`, database/table discovery | ok | | Type mapping (int64→bigint, string→text, int32→int, double→double, `fixed_size_list<float,16>`→`array<float>`) | ok | | Full scan — 1024 rows, `sum(row_id) = 524800` | ok | | Substrait predicate pushdown (`lancePushdownPredicate`), non-convertible expression left as BE residual | ok | | Projection pruning | ok | | IVF_FLAT vector search — full probe reproduces flat search row-for-row on the exact `16·d²` ladder | ok | | Global Top-K across fragments (row 1024 lives in the second fragment) | ok | | `FOR VERSION AS OF` — after appending 10 rows: latest 1034, v2 1024, v1 1024 | ok | Working configuration, for reference: ```sql CREATE CATALOG lance_gvt PROPERTIES ( "type" = "lance", "lance.catalog.type" = "rest", "lance.rest.uri" = "http://127.0.0.1:9101/lance";, -- Gravitino's /lance base path "lance.namespace.parent" = "lance_catalog", -- Doris database == Gravitino schema "s3.endpoint" = "http://127.0.0.1:9000";, "s3.access_key" = "minioadmin", "s3.secret_key" = "minioadmin", "s3.region" = "us-east-1", "use_path_style" = "true" ); ``` Two notes for whoever picks this up: 1. `lance.rest.uri` has to carry the server's base path (`/lance` here). `LanceRestMetastoreProperties.validateRestUri` already permits a path component, and the client appends `/v1/…` to it, so this works — worth a line in the docs, since the in-repo stub is mounted at the root and never exercises it. 2. **Test coverage has a hole shaped exactly like Problem 1.** `test_lance_rest_catalog.groovy` does cover credential vending, but the stub it talks to vends `aws_`-prefixed keys (`lance_rest_server.py` returns `aws_access_key_id`, `aws_secret_access_key`, `aws_region`), so the unprefixed alias path — which is what a real server emits — is never exercised. Teaching the stub to vend the unprefixed spelling in one case would pin this. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
