andygrove opened a new pull request, #7664: URL: https://github.com/apache/datafusion-comet/pull/7664
## Which issue does this PR close? Part of #7362. ## Rationale for this change arrow-java 20.0.0 fixes several bugs Comet works around, listed in #7362. The fix that lets Comet drop code in this PR is apache/arrow-java#1119, which stops the zstd codec writing past its output buffer (#7305). This is a draft until 20.0.0 is released. CI cannot resolve the `20.0.0` artifacts until then. I tested it locally against artifacts built from the `v20.0.0-rc0` tag. ## What changes are included in this PR? - `arrow.version` goes from 18.3.0 to 20.0.0. - `arrow-compression` now depends on `at.yawk.lz4:lz4-java`, a fork of lz4-java with the same `net.jpountz` classes as the `org.lz4:lz4-java` that Spark ships. The duplicate-class enforcer rule fails on it, so it is excluded like the other codec libraries Comet takes from Spark. Arrow only calls lz4 APIs that `org.lz4` 1.8.0 already has. - `CometZstdCompressionCodec` is removed. Arrow 20's `ZstdCompressionCodec.doCompress` is the same fix, so the cache uses Arrow's codec directly. `CometZstdCompressionCodecSuite` becomes `ArrowZstdCompressionCodecSuite`, and its guarded-allocator test now checks Arrow's codec. The test that showed the old defect and the test of the removed error helper are gone. - `BitVectorHelper.getValidityBufferSize` was removed upstream. Comet calls `getValidityBufferSizeFromCount`, which computes the same size. - `setValueCount(0)` on a large-offset vector now allocates the initial offset. The `UtilsSuite` test that covers a vector with no offset buffer now uses a freshly constructed vector instead. User-facing change: a zstd failure while compressing a cached batch now surfaces as Arrow's `RuntimeException` rather than a Comet internal error. With the corrected capacity, zstd can only fail there by running out of memory. The `VectorLoader.loadBuffers` leak in #7362 is still present in 20.0.0, so `CachedBatchIpc.decompressed` stays. The `ArrowImporter` cleanup for apache/arrow-java#1239 is left for a follow-up. ## How are these changes tested? Existing tests. With arrow-java built from `v20.0.0-rc0` and installed locally, on macOS aarch64 with the default Spark 4.1 profile, these pass: `UtilsSuite`, the `org.apache.spark.sql.comet.execution.arrow` suites (including `ArrowZstdCompressionCodecSuite`), the native and columnar shuffle suites, `CometShuffleInputConversionSuite`, `CometExecSuite`, `CometArrayExpressionSuite`, the in-memory cache suites and the `org.apache.comet.parquet` suites. The S3 suites were not run because they need Docker. Other Spark profiles have not been run locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
