Yicong-Huang opened a new pull request, #6024:
URL: https://github.com/apache/texera/pull/6024
### What changes were proposed in this PR?
`PveResourceSpec` runs in the `amber` unit CI job
(`AMBER_TEST_FILTER=skip-integration`) but is not tagged `@IntegrationTest`, so
every run shelled out to **real `pip` over the network**:
- each `createNewPve` ran `pip install -r requirements.txt` (the full amber
dependency set), and
- the lazy `resolveSystemPackages()` installed `requirements.txt` into a
throwaway venv, then `pip freeze`
— roughly 6 full installs per spec run. That made a PR-blocking unit spec
**network-dependent (flaky)** and **slow**.
This PR funnels every child process in `PveManager` (venv creation, pip
install / uninstall / freeze) through a single injectable seam:
```scala
private[pythonvirtualenvironment] var runProcess: ProcessRunner =
(command, env, logger) => Process(command, None, env: _*).!(logger)
```
Production wiring is unchanged (the default runner executes the command for
real). `PveResourceSpec` swaps in a fake that fabricates the
`<venv>/bin/{python,pip}` layout, emits canned output, and returns a
configurable exit code — so the spec is **fully hermetic: no venv, no pip, no
network**. `PveManager` still owns the metadata files (`user-packages.txt`) and
queue messages, so that logic stays under test.
Because failures are now cheap to trigger, this also adds negative coverage
that was previously impractical: venv-create failure, requirements-install
failure, user-package install failure, and system-package rejection.
| Before | After |
| --- | --- |
| ~6 real `pip install` per run, hits PyPI | 0 network calls |
| flaky on network hiccups, ~minutes | deterministic, ~8s |
| only happy-path assertions | + 4 negative/failure cases |
### Any related issues, documentation, discussions?
Closes #6023
### How was this PR tested?
`PveManager` is an `object`, so the test mutates the shared `runProcess` in
`beforeAll` and restores it in `afterAll`; the lazily-resolved system package
set is forced once up front so its cached value is independent of test order.
Run with JDK 17:
```bash
STORAGE_JDBC_USERNAME=texera STORAGE_JDBC_PASSWORD=password \
sbt "WorkflowExecutionService/testOnly
org.apache.texera.web.resource.pythonvirtualenvironment.PveResourceSpec"
# -> Tests: succeeded 25, failed 0, in ~8s
```
Also green:
```bash
sbt scalafmtCheckAll
sbt "WorkflowExecutionService/scalafixAll --check"
```
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
