FANNG1 opened a new pull request, #13929:
URL: https://github.com/apache/gravitino/pull/13929
### What changes were proposed in this pull request?
Upgrades the Lance dependencies:
- `lance-core`: `6.0.0` -> `12.0.0`
- `lance-namespace-core` / `lance-namespace-apache-client`: `0.7.5` ->
`0.11.1`
No production Java change is required. Gravitino does not implement
`org.lance.namespace.LanceNamespace` (its REST layer is hand-written JAX-RS
over its own `lance-common` ops interfaces), so 0.11.1's interface changes
(`namespaceExists`/`tableExists` now returning response objects,
`countTableRows`, `queryTable`) do not reach it. Every generated-client method
Gravitino calls is signature-identical, no model class was removed, and its
`lance-core` usage is all builder APIs that survived 6.0.0 -> 12.0.0.
Supporting changes:
- **OpenTelemetry excluded.** `lance-core` 12.0.0 adds compile-scope
`io.opentelemetry:opentelemetry-api`, used only by the opt-in
`org.lance.otel.LanceMetrics` bridge that Gravitino never enables. Excluding it
in `core` keeps the distribution and the legal files unchanged.
- **`lance-spark` matrix.** Adds `lance-spark-bundle` 0.8.0 (the bundle
built on lance 12.0.0 / namespace 0.11.1) and makes it the primary bundle, plus
the two CI workflow defaults that duplicate the list.
- **Blob file-format tests.** Lance's default file format version is now
2.2, so the blob tests no longer rely on that default and pin the version
explicitly via the blob column that selects it.
- Docs, the compatibility matrix, and stale version comments.
### Why are the changes needed?
To keep the Lance REST service consistent with the Lance community. 12.0.0 +
0.11.1 is the combination current Lance clients ship with —
`lance-spark-bundle` 0.8.0 embeds exactly that pair.
Fix: #13803
### Does this PR introduce _any_ user-facing change?
Yes, two:
1. **Lance's default file format version is now 2.2** (it was below 2.2 with
`lance-core` 6.0.0). A table created without blob columns therefore accepts a
later `External("lance.blob")` column but rejects
`External("lance.blob.legacy")`; previously it was the other way round.
Existing tables keep their stored version — an upgrade does not rewrite it.
Documented in `docs/lakehouse-generic-lance-table.md`.
2. **The server now accepts `lance-namespace` 0.11.1 request fields.** The
new optional `branch` and `tag` fields are accepted and ignored, rather than
rejected as unknown fields. Responses carry an empty `context` map. Branch,
materialized-view, `update_field_metadata` and `rename` endpoints remain
unimplemented.
No API or property-key changes.
### How was this patch tested?
All of the following pass locally (macOS, JDK 17):
- `./gradlew :lance:lance-common:test :lance:lance-rest-server:test
-PskipDockerTests=true` — 283 tests, including all 6 ITs
- `./gradlew :lance:lance-rest-server:lanceSparkMatrixTest
-PskipDockerTests=true` — 14 tests against each of `0.2.0`, `0.4.0`, `0.5.1`,
`0.8.0`, so both newer and older clients are covered
- `./gradlew :catalogs:catalog-lakehouse-generic:test
-PskipDockerTests=true`, including `CatalogGenericCatalogLanceIT`
- `./gradlew :core:coreUnitTest`, `./gradlew :spark-connector:spark-3.5:test`
- `./gradlew compileDistribution -PskipWeb=true` then
`:lance:lance-rest-server:test --tests "*LanceRESTServiceIT" -PtestMode=deploy`
— the deploy-mode run is what validates the OpenTelemetry exclusion against the
packaged classpath
- `:clients:client-python:lanceRayMatrixTest` with `lance-ray` 0.4.2 — the
unchanged Python client pins still work against the 0.11.1 server
- `./gradlew spotlessCheck`
New tests:
`TestLanceNamespaceOperations#testDescribeTableAcceptsNewOptionalRequestFields`
(posts raw JSON carrying `branch`/`tag`, which the pre-upgrade server would
have rejected) and three rewritten blob file-format tests in
`TestLanceTableOperations` covering both rejection directions and the new
default-version behaviour.
Two things worth flagging in review: the distribution grows ~30 MB
(lance-core 12's jar is 201 MB vs 172 MB), and the matrix job now downloads
four bundles (~1.1 GB) and runs ~33% longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
