Yicong-Huang opened a new pull request, #6317: URL: https://github.com/apache/texera/pull/6317
## What The four sbt-based CI jobs — `amber`, `amber-integration`, `platform`, and `platform-integration` — each compile the shared common-module graph (DAO, WorkflowCore, WorkflowOperator, Auth, Config, …) from scratch. That is up to **14 runner legs** redoing the same Scala build every run. This adds a single **`scala-build`** prebuild job that compiles the whole Scala world **once** and hands the result to the downstream jobs as artifacts. ## How `scala-build` (ubuntu, one leg) runs one sbt invocation: - `scalafmtCheckAll` + `scalafixAll --check` (cross-cutting lints, moved here from `amber` / `amber-integration`) - all 7 `<svc>/dist` - root `Test/compile` (compiles every module's test scope; `TexeraProject` aggregates all modules) …then uploads two artifacts: - **`scala-build-target`** — per-module `target/` tree (classes, test-classes, Zinc analysis, generated JOOQ sources, dists). Restored by `amber` / `amber-integration` (Linux) / `platform` so their `jacoco`/`test` reuse compilation. - **`scala-dists`** — just the dist zips, for `platform-integration`'s smoke-boot. Downstream jobs drop their own lint + compile and consume the artifacts. ## Respecting the label gate `scala-build.if` is the **union of the four Scala `run_*` inputs**. So a docs-only / frontend-only PR never runs it, and — since each downstream Scala job keeps its own `if: run_*` — every Scala job stays skipped in lockstep. The `required-checks` aggregator depends on the whole `build` reusable-workflow result, so the new job is transparent to the merge gate (no `.asf.yaml` change). Artifact names are computed per `build.yml` invocation (from `job_name_suffix` / `backport_target_branch`) so the main `build` and the `backport` legs — which run in one workflow run — don't collide on upload (v4 artifact names are unique per run). ## Goal & how to read the CI results Primary goal is **deduplicating compilation / cutting runner-minutes**, not necessarily wall-clock. A serial prebuild + artifact upload/download is inserted before the fan-out, so wall-clock may stay flat or move slightly either way. **This PR is to gather real numbers** — please compare wall-clock and total runner-minutes against a baseline before we commit to the approach. ## Known caveats / follow-ups - **macOS `amber-integration` still self-compiles** — Zinc analysis embeds OS-specific absolute paths and is not portable across runners. Because `needs:` is per-job (not per-matrix-leg), the macOS leg now *waits* for `scala-build` before starting its own compile, which adds latency to that leg specifically. If the measured hit is large, split it into an independent job with no `needs:`. - **Zinc cross-job reuse** relies on `github.workspace` being a stable path across GitHub-hosted ubuntu runners. If the analysis is ever rejected, sbt just recompiles — slower, never wrong. Worth confirming from the logs that downstream `jacoco`/`test` do **not** trigger a full recompile. - **Artifact size / download fan-out** — the 6-leg `platform` matrix each downloads `scala-build-target`. Watch the download time; if it dominates, slim the tar or split per service. Opened as **draft** to observe CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
