u70b3 opened a new pull request, #67235: URL: https://github.com/apache/doris/pull/67235
### What problem does this PR solve? Issue Number: #66497 Related PR: #66637 (merged), #67201 (PR3A, open — independent; this PR shares no files with it), #66671 (open — independent) Problem Summary: This is the second sub-PR (PR3B) of delivery slice 3 of the v5.1 design ([final 4.2 contract](https://github.com/apache/doris/issues/66497#issuecomment-5301314544), scope confirmed in [this review](https://github.com/apache/doris/issues/66497#issuecomment-5301637401)): the durable Lance index job infrastructure — job records, the compact dual-state lifecycle, the same-name fence, three-level unresolved quotas, typed results, and replay semantics — behind a master-owned manager. **This PR has no user-visible entry point**: no SQL, no admission, no dispatcher, no worker, no enablement gate. It is the persistence and state-machine foundation that PR3C (admission + job SQL) and PR3D (dispatch + fake-worker fault tests) build on. What this PR adds: - The minimal durable job record of Section 7.2 (`LanceIndexJob`): job identity/creator/revision/bounded timestamps, persisted target identity and same-name fence key material (catalog id, `DIRECTORY` provider tag, normalized dataset locator, display + normalized index name), mutation intent, admitted dataset version and the ordered schema-contract-v1 representation (Section 4.2), the independent mutation/refresh states, the typed result with bounded sanitized message, dispatch identity (backend id, BE process epoch, immutable invocation id, deadline), possible-live ownership with termination proof, and the FORCE audit fields (populated by PR3E; only replay semantics land here). The record carries no credentials and no unbounded values (Section 4.3/8). - The Section 6.1/6.2 dual state machines and the Section 6.3 provider-result classification table as data plus one pure `classify` function (13 typed result codes × CREATE/REPLACE/DROP × IF-flags × external-advancement), including `IF_CONDITION_NOOP` only for `DROP IF EXISTS` + `LANCE_ERR_NOT_FOUND`. Normalization v1 (Section 4.1) for index names (`toLowerCase(Locale.ROOT)`) and dataset locators (scheme case, trailing slashes, credential-bearing URL rejection); the fence key and the three-level (table-locator/catalog/global) unresolved-quota counters of Section 5.4. - `LanceIndexJobManager` (Appendix B seam: a master-owned minimal job/fence manager; it deliberately reuses neither the generic scheduling `JobManager` nor internal `IndexChangeJob`, neither of which provides external one-shot CAS, no-redispatch, same-name fence, or possible-live semantics). All durable transitions share one write-path shape — validate under the write lock (state legality, revision CAS, callback identity), append one upsert record to the edit log, then apply the same record locally — so master and followers run identical apply logic. Fence and unresolved quota live and die together exactly as Sections 5.4/6.4 require: held by PENDING/RUNNING, by terminal jobs until their required refresh is DONE, and by UNKNOWN until a durable FORCE_RELEASE; a quota or fence rejection precedes any durable write, leaving no job, no fence, and no record. - Replay per Section 7.3. `replayUpsertJob` is a verbatim replace with a monotonic-revision guard and performs no state transformation — a follower tailing a live master must keep a fresh RUNNING record RUNNING. The RUNNING→UNKNOWN transition happens only in the master-election sweep (`Env.transferToMaster`, after metadata replay and before any master daemon starts, mirroring the `insertOverwriteManager.allTaskFail()` precedent), which writes UNKNOWN upserts through the same identity-checked channel so followers converge and stale callbacks (revision/invocation/epoch mismatch) are rejected. Refresh RUNNING is downgraded to REQUIRED at the sweep so the idempotent external-table refresh can resume; replay never calls lance-c again. - Persistence wiring per Doris convention: one new edit-log op (`OP_LANCE_INDEX_JOB_UPSERT = 500`), `JournalEntity`/`EditLog` dispatch, a new `lanceIndexJobManager` image module appended to `PersistMetaModules.MODULE_NAMES` (no `FeMetaVersion` bump; old images simply never invoke the load method, and `Env` pre-initializes an empty manager). Explicitly not in this PR: SQL/admission/IF preflight and job SQL (PR3C, including the Section 9.7 gating configuration — quota limits here are parameters, not Config); BE selection, dispatch, possible-live slot reservation on BEs, and worker invocation (PR3D); the `FORCE_RELEASE` transition and its auth protocol (PR3E — only the durable fields and the replay row exist); Arrow-schema→contract construction (PR3C admission); job-record retention/GC (follows the bounded-retention policy with PR3E). PR1's `SHOW INDEX` and PR2's inspection surface are untouched; this branch shares no files with PR3A. ### Release note None (internal infrastructure only; no user-visible behavior change). ### Check List (For Author) - Test - [ ] Regression test - None: this PR adds no SQL surface and no external behavior; regression suites for the DDL surface belong to PR3A (#67201) and the end-to-end lifecycle to later slices. - [x] Unit Test - New suites under `org.apache.doris.datasource.lance.job` (105 cases, all green with checkstyle enabled): normalization v1 (incl. the Turkish-İ corner), locator normalization and rejection forms, the full Section 6.3 classification matrix (independently restated per cell), state-machine legality (UNKNOWN has no outgoing transitions; refresh independence; revision CAS), fence/quota co-release timing, quota three-level boundaries, the Section 7.3 replay matrix (PENDING re-dispatchable once; RUNNING swept to UNKNOWN and never redispatchable; terminal jobs resume only refresh; UNKNOWN rebuilds fence/quota/possible-live; force-released UNKNOWN frees the name), stale-callback rejection, replay idempotence and monotonic revision, corrupt-record tolerance, manager image write/read round-trip with derived-state rebuild, and `JournalEntity` op-500 round-trip. - Scoped regression green (469 cases): `org.apache.doris.persist.**`, `org.apache.doris.journal.**`, `org.apache.doris.dictionary.**`, `org.apache.doris.indexpolicy.**`, `org.apache.doris.job.**`, `org.apache.doris.datasource.lance.**` (`cd fe && mvn test -pl fe-common,fe-core -am -DfailIfNoTests=false -Dtest='...'`, no `-Dcheckstyle.skip`). - [x] Manual test - `mvn compile -pl fe-common,fe-core -am` green with checkstyle (validate phase); the new image module binding resolves at `PersistMetaModules` static init; the edit-log op code 500 verified unique repo-wide. - Behavior changed: - [x] No. New code paths are unreachable from any SQL or RPC surface in this PR; existing edit-log ops, image modules, and manager behaviors are unchanged. - Does this need documentation? - [x] No. User documentation tracks the broader lifecycle work in #66497 once mutation is enabled. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
