GitHub user Xiao-zhen-Liu edited a comment on the discussion: Design and merge plan: operator output port result cache (MVP)
@chenlica @Yicong-Huang Following our October 1 discussion (where #8731 to #8734 were put on hold to examine @Yicong-Huang's proposal and how the design extends past full reuse), here is a summary of what we found and the two choices to settle. ### 1. Any reuse method needs a workflow-wide backward pass At the October 1 meeting, @Yicong-Huang asked whether reuse can be decided inside the cost estimator and resource allocator by assigning 0 workers to operators with saved results, without a separate pass before scheduling (#8731). When `B` has a saved result and upstream `A` does not, checking per operator wastes compute on `A` (left), while propagating only within one region `R1` skips `A` even when a second region `R2` still needs `A`'s output (right):  ### 2. Where that global pass lives, and why it separates cleanly from Ramen (#3605) In `CostBasedScheduleGenerator`, regions are formed across the whole workflow first, and `CostEstimator` and `ResourceAllocator` (including planned Ramen, #3605) are called afterward on **one region at a time**. Running the backward pass before region formation lets Ramen allocate normal running regions with no changes to Ramen:  ### 3. Why skipped operators form Level 0 skip regions instead of staying in running regions Even after a workflow-wide pass decides what skips, keeping 0-worker `B` inside a running region creates a false 60 s completion barrier (and breaks multi-output `Split` when only one port is saved, #8731, #8732). Putting skipped operators in Level 0 skip regions (#8733) lets `A` and `C-Build` run in parallel and unblocks downstream regions as soon as `C-Build` finishes:  ### 4. Extending beyond Eager Reuse: Two-Layer Search vs. One-Layer Search When a downstream saved result (`D`) sits on slow storage while an upstream saved result (`B`) sits on fast storage, recomputing `C -> D` and pipelining can beat Eager Reuse. Both **Two-Layer Search** (wrapping `main`'s 2-state scheduler) and **One-Layer Search** (replacing it with a 4-state link queue) build on **#8731 to #8734 unchanged**:  ### 5. Two framework choices to settle Because open PRs **#8731 to #8734** (along with the execution, runtime-statistics filter, storage, and editor UI PRs) are shared unchanged across every option, aligning on two choices unblocks the merge plan: 1. **Rollout scope:** merge the **Two-Phase Eager Reuse MVP first** and add cost-based search next, **or** land **cost-based reuse search from the start** before enabling cache reuse. 2. **Search architecture:** **Option A (Two-Layer Search)** wrapping `main`'s unchanged 2-state `CostBasedScheduleGenerator`, **or** **Option B (One-Layer Search)** replacing the 2-state loop with the 4-state link queue.  <details> <summary>Additional October 1 details (intact PhysicalPlan, no SKIPPED state, runtime-statistics history filter, planned Peanut)</summary> | Question from October 1 | How the design handles it | | --- | --- | | Keep `PhysicalPlan` intact for downstream lookups | Scheduler keeps full `PhysicalPlan`; only retained operators enter region search (#8731, #8733). | | No new `SKIPPED` operator or region state | Reused operators finish as `COMPLETED` with `reused_from_cache = true` (#6729) and saved row counts from `operator_port_cache` (#5967). | | Do not skew `CostEstimator` / Ramen runtime history | Persist `reused_from_cache` in runtime statistics and ignore `reused_from_cache = true` rows so 0-worker, 0.0 s runs never overwrite compute history. | | Compatibility with planned Peanut (`U_train -> U_predict`) | Planner and search operate on the expanded `PhysicalPlan`, so mini-operators from planned Peanut are scheduled and cached like any other operators. | </details> GitHub link: https://github.com/apache/texera/discussions/5880#discussioncomment-19505621 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
