andygrove opened a new pull request, #7664:
URL: https://github.com/apache/datafusion-comet/pull/7664

   ## Which issue does this PR close?
   
   Part of #7362.
   
   ## Rationale for this change
   
   arrow-java 20.0.0 fixes several bugs Comet works around, listed in #7362. 
The fix that lets Comet drop code in this PR is apache/arrow-java#1119, which 
stops the zstd codec writing past its output buffer (#7305).
   
   This is a draft until 20.0.0 is released. CI cannot resolve the `20.0.0` 
artifacts until then. I tested it locally against artifacts built from the 
`v20.0.0-rc0` tag.
   
   ## What changes are included in this PR?
   
   - `arrow.version` goes from 18.3.0 to 20.0.0.
   - `arrow-compression` now depends on `at.yawk.lz4:lz4-java`, a fork of 
lz4-java with the same `net.jpountz` classes as the `org.lz4:lz4-java` that 
Spark ships. The duplicate-class enforcer rule fails on it, so it is excluded 
like the other codec libraries Comet takes from Spark. Arrow only calls lz4 
APIs that `org.lz4` 1.8.0 already has.
   - `CometZstdCompressionCodec` is removed. Arrow 20's 
`ZstdCompressionCodec.doCompress` is the same fix, so the cache uses Arrow's 
codec directly. `CometZstdCompressionCodecSuite` becomes 
`ArrowZstdCompressionCodecSuite`, and its guarded-allocator test now checks 
Arrow's codec. The test that showed the old defect and the test of the removed 
error helper are gone.
   - `BitVectorHelper.getValidityBufferSize` was removed upstream. Comet calls 
`getValidityBufferSizeFromCount`, which computes the same size.
   - `setValueCount(0)` on a large-offset vector now allocates the initial 
offset. The `UtilsSuite` test that covers a vector with no offset buffer now 
uses a freshly constructed vector instead.
   
   User-facing change: a zstd failure while compressing a cached batch now 
surfaces as Arrow's `RuntimeException` rather than a Comet internal error. With 
the corrected capacity, zstd can only fail there by running out of memory.
   
   The `VectorLoader.loadBuffers` leak in #7362 is still present in 20.0.0, so 
`CachedBatchIpc.decompressed` stays. The `ArrowImporter` cleanup for 
apache/arrow-java#1239 is left for a follow-up.
   
   ## How are these changes tested?
   
   Existing tests. With arrow-java built from `v20.0.0-rc0` and installed 
locally, on macOS aarch64 with the default Spark 4.1 profile, these pass: 
`UtilsSuite`, the `org.apache.spark.sql.comet.execution.arrow` suites 
(including `ArrowZstdCompressionCodecSuite`), the native and columnar shuffle 
suites, `CometShuffleInputConversionSuite`, `CometExecSuite`, 
`CometArrayExpressionSuite`, the in-memory cache suites and the 
`org.apache.comet.parquet` suites. The S3 suites were not run because they need 
Docker. Other Spark profiles have not been run locally.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to