andygrove opened a new issue, #6916: URL: https://github.com/apache/datafusion-comet/issues/6916
### What is the problem the feature request solves? LZ4 compression of native shuffle blocks (the `lz4_flex` frame encoder in `ShuffleBlockWriter::compress_ipc_stream`) is the most expensive single function in the profile of `CometIcebergEndToEndBenchmark` (#6910): 11.9% of all CPU for nested types and 11.5% for primitives. LZ4 is already the fastest of the supported codecs. Codec A/B on the fully native configuration, average statement time: | Codec | Nested types | Primitives | Shuffle bytes (nested) | | --- | --- | --- | --- | | `lz4` (default) | 1198 ms | 546 ms | 344 MB | | `snappy` | 1237 ms | 573 ms | 371 MB | | `zstd` (level 1) | 1301 ms | 569 ms | 220 MB | | none (`spark.shuffle.compress=false`) | 1074 ms | 498 ms | 524 MB | Disabling compression is about 10% faster end to end on a local disk, so a cheaper compression step has room to pay off before the bytes saved on a network do. ### Describe the potential solution - Skip compressing data that does not compress, for example by compressing per IPC buffer and storing a buffer raw when compression does not shrink it, or by sampling the ratio once per partition. - Measure the C LZ4 library (`lz4-sys`) against `lz4_flex` for compression throughput. - Reuse encoder state across batches rather than building a frame encoder for every block. ### Additional context `compress_ipc_stream` in `native/shuffle/src/writers/shuffle_block_writer.rs`. arrow-rs #8423 discussed reusing `FrameEncoder` for IPC compression. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
