morningman opened a new pull request, #69575:
URL: https://github.com/apache/doris/pull/69575
### What problem does this PR solve?
Issue Number: None
Related PR: #65730, #68781
Original author: @u70b3 (the op 500 and image-module tombstones follow
branch-4.1 #68781; the commit carries Co-authored-by)
Problem Summary:
**In short.** This PR makes master read the metadata that branch-4.1's Lance
catalog (#65730, released in 4.1.4) and its since-removed index job framework
(tombstoned by #68781) wrote: an FE image written by 4.1.4 or later with a
Lance catalog in it now loads. Before this PR the image failed to parse, so a
cluster with a Lance catalog could not be upgraded from any released 4.1.4 /
4.1.4.1 / 4.1.5 build (or from 4.2) to a master-based build: the FE did not
start. The Lance catalog's GSON labels are mapped to the plugin-driven classes,
as the Iceberg, Paimon and LakeSoul labels already are, and the catalog stays
registered, unusable until a Lance connector plugin is installed. The two
pieces of metadata that only unreleased branch-4.x builds wrote (the Lance
index job edit-log record and image module) are kept readable as tombstones, as
branch-4.1 #68781 does.
**Background.** branch-4.1 added a Lance catalog as fe-core classes in
#65730 (released in 4.1.4). Like every external catalog, it is persisted in the
image's `datasource` module as GSON JSON whose `clazz` field names the class,
`LanceExternalCatalog`, with `logType` `LANCE`. master has no Lance classes:
external catalogs are served by connector plugins, and the labels of the
catalogs that used to be fe-core classes are redirected to
`PluginDrivenExternalCatalog` / `PluginDrivenExternalDatabase` /
`PluginDrivenMvccExternalTable` by `registerCompatibleSubtype` in `GsonUtils`.
A Lance connector plugin does not exist on master yet; it will be ported later.
branch-4.1 also had, for a while, a durable Lance index job framework
(#67235) that wrote edit-log op 500 (`OP_LANCE_INDEX_JOB_UPSERT`) and an image
module named `lanceIndexJobManager`. #68781 removed it before any release.
4.1.5 and branch-4.2 keep op 500 as a tombstone that is read and discarded, and
list the module as deprecated, so the journals and images of the builds in
between stay readable.
The upgrade fixtures on master (`upgrade/413`) were generated from 4.1.3,
before Lance existed, so no test caught this.
**The problem, and what it cost.** Compared on master against fixtures
emitted by real 4.1.5 bytecode:
| Metadata from a 4.1.4+ FE | master before this PR | after this PR |
|---|---|---|
| image whose `datasource` module has a Lance catalog (every 4.1.4+ cluster
with one) | image fails to parse: `cannot deserialize interface
org.apache.doris.datasource.CatalogIf subtype named LanceExternalCatalog; did
you forget to register a subtype?`; the FE cannot start | loads; the catalog is
a `PluginDrivenExternalCatalog` of type `lance` with all its properties,
comment and auto-analyze policies; using it fails with `No ConnectorProvider
found for plugin-driven catalog: <name>, type: lance. Ensure the connector
plugin is installed.` |
| `OP_CREATE_CATALOG` of a Lance catalog | replays as a degraded catalog
(the entry carries only properties) | unchanged |
| op 500 (only from unreleased branch-4.x builds) | replay fails: `UNKNOWN
Operation Type 500` | read and discarded |
| image module `lanceIndexJobManager` (only from unreleased branch-4.x
builds) | image fails: `Unknown meta module: lanceIndexJobManager` unless
`ignore_unknown_metadata_module = true` | skipped as deprecated |
The only labels in the GSON registries of 4.1.4.1, 4.1.5 and 4.2.0-rc01 that
master does not know are `LanceExternalCatalog`, `LanceExternalDatabase` and
`LanceExternalTable`.
**How this PR fixes it.**
- `GsonUtils`: `LanceExternalCatalog` → `PluginDrivenExternalCatalog`,
`LanceExternalDatabase` → `PluginDrivenExternalDatabase`, `LanceExternalTable`
→ `PluginDrivenMvccExternalTable` (`LanceExternalTable` implemented
`MvccTable`). 4.1.x never persists an external database or table, so only the
catalog label can actually appear; the other two complete the registry, as for
the other migrated connectors.
- `InitCatalogLog.Type` keeps `LANCE`, so the `logType` 4.1.4+ wrote still
deserializes to itself. Nothing depends on it: a Lance catalog always persists
its `type` property (no resource type creates one), and
`PluginDrivenExternalCatalog#gsonPostProcess` rewrites the logType to `PLUGIN`.
- `OperationType.OP_LANCE_INDEX_JOB_UPSERT = 500` is reserved:
`JournalEntity` reads its Text-framed JSON and discards it, `EditLog` replays
nothing. `lanceIndexJobManager` joins
`PersistMetaModules.DEPRECATED_MODULE_NAMES`. Both follow branch-4.1 #68781.
Until a Lance connector plugin is installed, an upgraded cluster lists its
Lance catalogs and fails only when one is used; installing the plugin makes
them work without any metadata change, since the catalog properties are handed
to the connector unchanged.
**Results.**
- New golden fixtures `fe/fe-core/src/test/resources/upgrade/415/`, emitted
by 4.1.5-release bytecode like `upgrade/413` (generator and `PROVENANCE.txt`
included, byte-deterministic): the `datasource` module with three Lance
catalogs (filesystem on S3, filesystem on `file://`, REST with bearer auth),
their `OP_CREATE_CATALOG` entries, and 4.1.5's label registries.
- `Legacy415LanceUpgradeTest` (9 tests): the image loads and every Lance
catalog becomes a `PluginDrivenExternalCatalog` of type `lance` with the
`PLUGIN` logType; every persisted property, the comment, `lastUpdateTime` and
the auto-analyze policy survive; the persisted logType names still deserialize;
without a plugin the catalog fails only when used, with the message above; a
provider claiming `lance` receives exactly the persisted properties, from the
image and from the journal; the journal entries replay with and without a
plugin; an op 500 record followed by a real 4.1.5 entry is consumed exactly,
and replaying it needs no `Env`; `lanceIndexJobManager` stays deprecated.
- `LanceGsonCompatReplayTest` (4 tests): the three labels replay as the
expected classes (the table as the MVCC variant), and they are the only labels
4.1.5 registered beyond 4.1.3.
- Each of these changes, undone on its own, turns tests red: dropping the
catalog label (5 tests, with the parse error above), the database label,
mapping the table label to the non-MVCC class, removing `LANCE`, the
`JournalEntity` case (`UNKNOWN Operation Type 500`), the `EditLog` case (the
replay ends in `System.exit(-1)`, as on a real FE) or the deprecated-module
entry.
- 127 tests of the two new classes and 17 related upgrade, GSON and journal
suites (`Legacy413*`, `*GsonCompat*`, `CatalogMgrTest`, `ExternalCatalogTest`,
`ConstraintPersistTest`, `BDBJEJournalTest`, ...) pass; fe-core checkstyle
passes.
### Release note
Fix: an FE upgraded from 4.1.4 or later (or 4.2) to this version loads an
image that contains a Lance catalog. The catalog is kept but cannot be used
until a Lance connector plugin is installed.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
An image with a Lance catalog loads instead of failing; journals with
op 500 and images with the `lanceIndexJobManager` module are read instead of
failing.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
