aglinxinyuan opened a new pull request, #6132:
URL: https://github.com/apache/texera/pull/6132
### What changes were proposed in this PR?
Add `MultiRegionWorkflowIntegrationSpec`, an end-to-end integration test for
a workflow that executes across **multiple regions**.
Cross-region data delivery goes through input-port materialization reader
threads (which replaced the cache-source operators in #3425). Until now that
path had no end-to-end coverage — only scheduling unit tests that count regions
on a physical plan. This spec adds that coverage with a big "X"-shaped workflow:
```
csvLeft ─┬─▶ join.build (port 0) ─┐
│ join ─▶ pythonSort ─▶ (terminal)
csvRight ─┼─▶ join.probe (port 1) ─┘
│
csvLeft ─┼─▶ union ───────────────────────────────────▶ (terminal)
csvRight ─┘ (two links fan into union's single port)
```
The hash join's probe-depends-on-build ordering forces the build input to be
materialized, cutting the plan into ≥2 regions whose boundary is crossed by the
reader-thread path. The test asserts both structure and correctness:
- the plan spans **≥2 regions** (via the same `CostBasedScheduleGenerator`
the controller uses at runtime);
- the self-join on the unique `Order ID` key yields **100 rows**, the Python
UDF returns them **sorted**, and the union concatenates both sources into **200
rows**.
A Python UDF (sort) makes this a real end-to-end run with worker
subprocesses, so it is class-level `@IntegrationTest` tagged and routed to the
`amber-integration` CI job (which provisions Python deps); the lighter `amber`
job excludes it.
### Any related issues, documentation, discussions?
Lands ahead of #6126 (removal of the now-dead `CacheSourceOpExec` /
`OpExecSource` plumbing) so the multi-region path has end-to-end coverage
**before** that refactor. Requested by @Yicong-Huang during review of #6126.
### How was this PR tested?
- `sbt "WorkflowExecutionService/Test/compile"` — green
- `sbt "WorkflowExecutionService/Test/scalafmtCheck"
"WorkflowExecutionService/scalafixAll --check"` — clean
- The spec itself runs in the `amber-integration` CI job (it spawns Python
workers, so it cannot run on Windows locally). Verified passing there: the run
spawned the `WF*-HashJoinOpDesc-build`/`-probe`, `WF*-UnionOpDesc`, and
`WF*-PythonUDFOpDescV2` workers and all tests passed.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8 [1M context])
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
