aglinxinyuan opened a new pull request, #6212: URL: https://github.com/apache/texera/pull/6212
### What changes were proposed in this PR? Backport of apache/texera#6185 to `release/v1.2`: upgrade Apache Arrow `15.0.2 → 19.0.0` and the Netty family `4.1.x → 4.2.15.Final` as one coordinated move. `arrow-memory-netty` reaches into Netty's allocator internals, so the 4.1/4.2 line cannot split — a split throws `NoClassDefFoundError: io/netty/util/concurrent/ThreadAwareExecutor` on the Arrow Flight transport to the Python workers. Arrow 19 is the first release targeting the Netty 4.2 line, so the two move together. The standalone `flight-grpc` artifact (discontinued after 15.0.2) is dropped — its gRPC transport now lives inside `flight-core`. The Netty family is extracted into a shared `nettyDependencyOverrides` val applied to amber and the three Arrow-bundling platform services, and the Jackson core family is pinned to `2.18.8` in `workflow-core`/`workflow-operator` (Arrow 19 otherwise evicts `jackson-databind` past the `>= 2.18.0, < 2.19.0` range `jackson-module-scala` allows). **Update:** this PR also folds in the `computing-unit-managing-service` startup fix from apache/texera#6209 (issue #6206), so the Arrow upgrade lands on `release/v1.2` without ever breaking that service's startup. #6185 pinned `jackson-databind` in `workflow-core`/`workflow-operator` and in `file-service`/`workflow-compiling-service`, but not in `computing-unit-managing-service`, whose dist consequently bundled `jackson-databind-2.21.0.jar` next to `jackson-module-scala 2.18.8` and aborted at boot with `Scala module 2.18.8 requires Jackson Databind version >= 2.18.0 and < 2.19.0 - Found jackson-databind version 2.21.0`. The fix adds the `jackson-databind` override to that service too (matching `file-service`/`workflow-compiling-service`). Notes on how the backport was produced: - **Source changes were hand-applied, not cherry-picked.** `release/v1.2`'s `build.sbt` is structurally different from `main`'s (no `Resource` module or `NotebookMigrationService`, `asfLicensingSettings` rather than `commonModuleSettings`, hadoop 3.3.1, no log4j-2 bridges), so a raw cherry-pick of #6185's squash would drag in unrelated structure. Only the Arrow/Netty/Jackson semantics from #6185 + #6209 were ported; the `common/workflow-operator/build.sbt` hunk is byte-identical to #6185's. - **The 8 `LICENSE-binary`(-java) / `NOTICE-binary` manifests were regenerated from `release/v1.2`'s own resolved runtime classpath**, not copied from `main`. `main`'s manifests reflect deps this branch does not have (hadoop 3.4.3, the log4j-2 bridges), so their transitive tails differ. On this branch the cascade resolves to grpc `1.62 → 1.79.0`, protobuf-java `3.25 → 4.33.4` (+ new `protobuf-java-util`), the netty `netty-codec → netty-codec-base`/`netty-codec-compression` split, new `arrow-memory-netty-buffer-patch` and `org.jspecify:jspecify 1.0.0`, `guava → 33.5.0-android`; `flight-grpc` and `eclipse-collections` drop out. ### Any related issues, documentation, discussions? Backports apache/texera#6185 (merged to `main`). Also includes the fix from apache/texera#6209 for #6206 (startup regression introduced by #6185). No `release/*` label is added — this PR *is* the backport. ### How was this PR tested? The binary-licensing manifests were verified the same way CI checks them, against each module's resolved `Runtime/managedClasspath` on this branch (which equals the dist's third-party jar set): - `check_binary_deps.py jar --license-binary <manifest> <lib>` passes for all four services in both strict (nightly) mode and `--ignore-transitive-version` (PR) mode: `OK: 407 / 311 / 323 / 350 JVM jars match LICENSE-binary` for amber / file-service / workflow-compiling-service / computing-unit-managing-service. - `generate_notice_binary.py` reproduces each committed `NOTICE-binary` byte-for-byte (regenerate-and-diff is clean for all four), matching the CI NOTICE check. - `sbt export <Service>/Runtime/managedClasspath` resolves cleanly for all four services with Arrow 19 + Netty 4.2 (no eviction or resolution errors); `computing-unit-managing-service` now resolves `jackson-databind-2.18.8.jar` (was `2.21.0`). ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context]) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
