andygrove opened a new pull request, #7306: URL: https://github.com/apache/datafusion-comet/pull/7306
## Which issue does this PR close? No tracking issue. This is a dependency upgrade. ## Rationale for this change Comet pins arrow-java at 18.3.0. 19.0.0 is the latest release and the version Spark 4.2.0 ships. Comet shades Arrow, so it does not have to match Spark's version, but staying current keeps us on the upstream serialization and C Data fixes (for example GH-343, offset buffers for empty and nested arrays in IPC streams, which the Python UDF runner writes). The upgrade is not just a version change. Arrow Java 19 changes two behaviors that Comet depended on. ## What changes are included in this PR? - `pom.xml`: `arrow.version` 18.3.0 to 19.0.0. - `ArrowImporter`: Arrow Java 19 no longer closes the source `ArrowArray` struct after `ArrayImporter.importArray` moves its contents into the importer's copy (GH-765). `ArrowImporter` relied on that, so every imported column would leak its `ArrowArray` struct (128 bytes) in `CometArrowImportAllocator` per batch. It now closes the source itself, guarded by `isClosed()`. The error path comment is updated to match. - `Utils.scala`, `UtilsSuite.scala`: `BitVectorHelper.getValidityBufferSize` is replaced by `getValidityBufferSizeFromCount`. The two names never exist in the same release, so this cannot compile against 18.3. - `UtilsSuite`: `setValueCount(0)` on a large variable-width vector now allocates an 8-byte offset buffer. The empty-vector test needs a vector with no offset buffer, so it uses an unallocated one. apache/arrow-java#88 (C Data import ignores `ArrowArray.offset`) is still open in 19.0.0, so the workarounds in `ffi.md` stay. ## How are these changes tested? On the default Spark 4.1 profile, 18 suites that exercise Arrow ran: the Arrow writers and streams, `NativeUtil`, `UtilsSuite`, `CometBroadcastValueSuite`, native shuffle, the in-memory cache, Parquet read, the UDF bridge, codegen dispatch, the Python runner and the Arrow Python UDF suites. Result: 440 passed, 0 failed, 9 canceled. Without the `ArrowImporter` change, `NativeUtilSuite` (4 tests) and `CometBroadcastValueSuite` (1 test) fail with leaked allocator memory, and the empty-vector `UtilsSuite` test fails without its edit. The same three suites pass on 18.3.0 before the upgrade. I also built the shaded jar and checked it: only `org/apache/arrow/c/**` stays unshaded, as configured, and the compression service file is relocated. Not covered by this local run, so CI should decide: - The 9 canceled tests are the native Arrow Python UDF tests, which skip themselves because my native build lacked the `python-udf` feature. The PyArrow end-to-end path was not run. - Spark 3.4, 3.5, 4.0 and 4.2 profiles, and the Iceberg suites. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
