u70b3 opened a new pull request, #69145: URL: https://github.com/apache/doris/pull/69145
## What problem does this PR solve? Issue Number: #66497 (slice 3 of the synchronous execution design under review in https://github.com/apache/doris/issues/66497#issuecomment-6163872351) Related PRs: #68957 (sync series 1 — FE orchestration and RPC contract). **This PR is stacked on the head of #68957: the branch carries 16 commits, the last 8 belong to this PR.** It completes the backend side. Supersedes the worker halves of the closed async PRs #68668/#68669: the isolation core is carried over, the async reporting machinery is not. ## Problem Summary After #68957, admitted index mutations reach the backend but the RPC handler is still a `NOT_IMPLEMENTED_ERROR` stub. This PR makes the backend actually execute synchronous Lance index mutations in an OS-isolated worker process and return a classified result on the same connection: - Every mutation runs in a separate `doris_be --lance-worker` process behind cgroup v2 limits (memory / pids / oom.group), `pidfd` supervision, `PDEATHSIG`, a sanitized environment, and `PR_SET_DUMPABLE=0`; if isolation cannot be verified the request is cleanly rejected before launch (never launched ⇒ confirmed failure). - The single-send contract is preserved end to end: one dispatch, one response, no retries, no callback channel. The response is sent only after the whole execution domain (worker + descendants) has terminated and been reaped, so a response is itself the termination proof that lets the FE release its in-memory active-name entry; if the response is lost the FE entry still converges at the statement budget. - The budget is converted once at receive into a local monotonic deadline that is never reset by queueing or reconnects, with an explicit reservation (`skew 1s + term grace 10s + reap bound 11s`) so that kill and reap always finish inside the FE-side budget; requests that cannot fund the reservation plus a minimal runnable window are rejected at receive, and re-checked at dequeue (double reject). - Result classification stays in the FE (#68957's four channels) and is extended with worker phase and identity markers, both additive on the wire: `worker_phase` (`PRE_INVOCATION_REJECTED` / `NATIVE_RETURN` / `INDETERMINATE_AFTER_LAUNCH`), an invocation-id / BE-epoch echo that mismatches degrade to indeterminate, and `observed_latest_version`, which lets the FE invalidate its local table cache when the worker observed an advancement while rejecting (§6.4 of the design). Pre-invocation rejections carry worker-defined negative codes; native returns pass the Lance error code through verbatim. No component parses message text to classify. - Worker-side re-verification before any native call: the schema contract is independently recomputed on the pinned admitted version (60 golden fixtures locked from both the FE builder and the worker recomputation), `CREATE ... IF NOT EXISTS` / `DROP ... IF EXISTS` are honored against the pinned snapshot as authoritative no-ops, and `DROP` resolves the normalized name uniquely in the pinned physical index list and drops by the stored spelling once. - Capacity is owned by the backend: a bounded receive queue plus a small executor pool (two-state slots), an invocation-id dedup set honouring "one invocation id, one execution", and fixed-name metrics (queue depth, in-flight workers, five outcome counters). Storage credentials travel only inside the pipe frame to the worker and never reach logs, messages, or metrics. The mutation gate `enable_lance_index_mutation` stays off by default; nothing in this PR enables it. ## Design readings reviewers may want to confirm 1. **Clock assumption (mechanism a).** Cross-machine skew is a documented assumption (`skew margin 1s`); no new wire field for verifiable expiration. The BE converts the absolute wall-clock deadline to a monotonic local deadline once at receive and deducts the kill reservation before it, so domain termination provably precedes the FE socket timeout. 2. **Duplicate invocation ids.** A repeated id is treated as a protocol violation: logged, not executed, and answered `OK + phase=INDETERMINATE` (a non-OK answer would fabricate "never executed" evidence). Ids rejected at receive are recycled; admitted ids are retained for a bounded window. 3. **Bounds travel on the internal frame, not the wire.** The worker runs before any BE global state and cannot read config, so the admission bounds snapshot is injected by the service from BE config into the pipe frame; the wire request to the FE is unchanged. 4. **Client disconnect.** Follows the design's recommendation (bounded finish): a disconnected client does not cancel the worker; capacity is released only when the execution domain terminates. This is the one §11 decision still awaiting maintainer confirmation. 5. **Rolling upgrade.** An old BE still answers `NOT_IMPLEMENTED_ERROR` (non-OK ⇒ confirmed failure), identical to the stub behaviour in #68957; mixed clusters are safe. 6. **No explicit kill signal from the FE.** After dispatch the FE never asks the backend to accelerate termination; the backend's own kill ladder (starting at `deadline − 22s`) guarantees the domain is gone before the FE budget expires, so all outcome invariants hold without distinguishing KILL from disconnect. Local `file://` datasets additionally stay rejected as a class (no operator-assertion escape hatch yet) — the conservative reading of the design's `file://` restriction. ## Tests - BE unit tests: 108/108 green under a delegated cgroup v2 runner (real fork/exec isolation, real kernel OOM, pids storms, CDC reaper races, orphan descendants, supervisor kill ladder), including a cross-language lock that recomputes all 60 golden contracts through real lance-c pinned opens and matches the FE-built fixtures. Note for coverage review: the isolation-core suites (supervisor / service fork paths) skip loudly on runners without a writable delegated cgroup v2 parent, so their lines are covered only where the suite reports real runs, never by silently green skips. - FE unit tests: focused `Lance*` suites green (classification permutations incl. phase/identity/band-defense/unknown-phase fallback, contract pins for the new result fields, golden fixtures); the PR1 suites are unchanged — zero flips. - Full FE regression on the stacked head: 10468 tests. Every failing class either reproduces identically on the unmodified base commit (timezone/network/native environment classes) or passes when re-run on an idle machine (the full run shared the box with backend builds for six hours); none is related to Lance code. - Groovy suites: assertions are now typed (error code + classification branch, never backend message text) and fork on a two-leg probe, so delegated runners exercise real execution while non-delegated runners assert the isolation rejection; shared preloaded fixtures are never dropped. ## Deferred - Tracer / fault-injection / query-consumption suites rewritten for the synchronous contract (next PR in the series); the worker-side debug points (`hang`, `skip_result`) and invocation-id cgroup naming they need are already in place. - BTREE/BITMAP lifecycle coverage and user documentation; distributed build (§8 of the design); verifiable expiration mechanism (b). ## Release note None. `enable_lance_index_mutation` remains disabled by default. ## Check List - [x] I have created an issue or PR for this change (tracked in #66497) - [x] I have added unit tests / groovy regression suites covering this change - [x] I have passed the BE unit tests locally on a delegated cgroup runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
