aglinxinyuan opened a new issue, #6133:
URL: https://github.com/apache/texera/issues/6133
### Task Summary
Cross-region data delivery goes through **input-port materialization reader
threads** (which replaced the cache-source operators in #3425). This path has
**no end-to-end coverage** — it is exercised only by scheduling unit tests that
count regions on a physical plan, so a regression in the materialization
read/write path could pass CI unnoticed.
This matters right now because the now-dead cache-source plumbing is about
to be removed (#6125 / #6126). The multi-region path — exactly what the removed
cache sources used to serve — should get end-to-end coverage **before** that
refactor.
Add an integration test that drives a big "X"-shaped workflow across
multiple regions:
```
csvLeft ─┬─▶ join.build (port 0) ─┐
│ join ─▶ pythonSort ─▶ (terminal)
csvRight ─┼─▶ join.probe (port 1) ─┘
│
csvLeft ─┼─▶ union ───────────────────────────────────▶ (terminal)
csvRight ─┘ (two links fan into union's single port)
```
The hash join's probe-depends-on-build ordering forces the build input to be
materialized, cutting the plan into ≥2 regions whose boundary is crossed by the
reader-thread path. A Python sort UDF makes it a real end-to-end run with
worker subprocesses.
Assertions:
| Aspect | Expectation |
| --- | --- |
| Region count | plan spans **≥2 regions** (via the same
`CostBasedScheduleGenerator` the controller uses at runtime) |
| Join branch | self-join on the unique `Order ID` key → **100 rows**,
returned **sorted** by the Python UDF |
| Union branch | concatenation of both sources → **200 rows** |
Notes:
- Python-tagged (`@IntegrationTest`) so it runs in the `amber-integration`
CI job (which provisions Python deps); the lighter `amber` job excludes it.
- Should land **before** #6126, which is then rebased on top so its CI
re-runs this test against the removal.
### Task Type
- [x] Test / Coverage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
