mengw15 opened a new issue, #6047: URL: https://github.com/apache/texera/issues/6047
### Task Summary Integration-test support is currently hard-wired to the `amber` module only. Turn it into a **reusable mechanism any module can opt into**, collected by a single cross-module `integration` CI job — so integration specs live next to the code they exercise, in whatever module that is. **Trigger (from #6045):** the shared `VirtualDocumentSpec` base and three live-catalog iceberg specs exercise `common/workflow-core` code, so their specs ideally live there — but common has no integration-test capability today, and pulling common's test scope into amber (`WorkflowCore % "test->test"`) causes a classpath conflict (Hadoop → `com.sun.jersey` 1.19's JAX-RS 1.x `Response` shadows amber's `javax.ws.rs-api:2.1.1`, breaking `HuggingFaceModelResourceSpec`). #6045 left those specs stranded in `amber` as a stopgap. common/workflow-core is just the **first module to need this** — the same capability should be available to the other Scala modules (`*-service`, `WorkflowOperator`, `DAO`, `Auth`, …) as they grow integration coverage. ### Current state - The `@IntegrationTest` tag lives at `amber/src/test/integration/.../amber/tags/IntegrationTest.java` — in amber's package and source dir. - The wiring (`Test / unmanagedSourceDirectories += src/test/integration` + the `AMBER_TEST_FILTER` skip/only switch) is only in `amber/build.sbt`. - Only `amber` has a `src/test/integration` source set. - The `amber-integration` CI job runs only `WorkflowExecutionService/test` (integration-only) + amber Python integration tests — no other module's integration specs. ### Suggested direction Reverse the coupling: instead of dragging a module's test scope into `amber` (which is what causes the classpath conflict), give each module its **own** integration source set running on its **own** classpath. - Move the `@IntegrationTest` tag to a shared location (e.g. `common`) so any module can reference it. - Factor the integration source-set + `AMBER_TEST_FILTER` wiring into **shared, reusable sbt settings** (an AutoPlugin or a shared settings sequence) applied to each participating module — not copy-pasted per `build.sbt`. - Generalize the `amber-integration` CI job into an `integration` job that runs the integration-tagged specs of **every participating module**, not just `WorkflowExecutionService`. - First adopter: move the iceberg specs + `VirtualDocumentSpec` from `amber` back into `common/workflow-core/src/test/integration`. Other modules opt in incrementally as they add integration coverage. Why this avoids the #6045 conflict: each module's integration specs compile and run on that module's own classpath (common already has Hadoop as a legit main dep), so common's `jersey` 1.19 is never mixed into amber's `javax.ws.rs` 2.1 classpath. Follow-up to #6045; related to #6034. ### Task Type - [x] Refactor / Cleanup - [x] DevOps / Deployment / CI - [x] Testing / QA - [ ] Documentation - [ ] Performance - [ ] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
