JonSnow1807 opened a new pull request, #7606: URL: https://github.com/apache/datafusion-comet/pull/7606
## Which issue does this PR close? Part of #6916. ## Rationale for this change `compress_ipc_stream` builds a new `lz4_flex` `FrameEncoder` for every shuffle block. That is a zeroed 16 KiB hash table, a 64 KiB input buffer and a 72,109 byte output buffer per block, plus a copy of the IPC stream into the input buffer. lz4_flex can’t reuse one encoder across writers because the encoder owns its writer. ## What changes are included in this PR? - New `lz4_frame.rs`: a small LZ4 frame writer on top of `lz4_flex::block::compress_into_with_table` that keeps its buffers and hash table across blocks. It writes standard LZ4 frames with independent 64 KiB blocks, the same layout remote shuffle already used. - `ShuffleCodecContext` holds the encoder. `release_zstd` is renamed to `release_workspaces` and drops both. - Local LZ4 always uses 64 KiB blocks now. Before, the block size followed the schema message, so very wide schemas got 256 KiB blocks. The block format doesn’t change, and all readers are native, so old and new writers and readers can be mixed. ## How are these changes tested? - New unit tests: single-block frames are byte-identical to lz4_flex’s `FrameEncoder`, and multi-block frames round trip through `FrameDecoder` for several sizes and write patterns. Another test covers releasing the encoder on the remote path. - The reference `lz4` CLI (1.9.4) decodes frames taken from real shuffle blocks. - `cargo test -p datafusion-comet-shuffle` passes, and clippy is clean. Benchmarks (`shuffle_writer`, Threadripper PRO 5975WX, 5 alternating rounds, median): | Benchmark | main | this PR | change | |---|---:|---:|---:| | write encoded, 8192 rows, LZ4 | 279.5 µs | 213.2 µs | -23.7% | | ipc_context mixed/8192/LZ4/reused | 282.3 µs | 214.4 µs | -24.1% | | ipc_context flat/8192/LZ4/reused | 226.1 µs | 191.0 µs | -15.5% | | end to end, 2000 partitions, LZ4 (cases from #7199) | 12.39 ms | 11.52 ms | -7.0% | | end to end, 8000 partitions, LZ4 (cases from #7199) | 23.69 ms | 22.37 ms | -5.6% | At 200 partitions, and for 128-row blocks with a fresh context (what remote shuffle does), the change is within noise. Allocations per 128-row LZ4 block drop from 171,325 bytes to 17,296, the same as no compression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
