andygrove opened a new issue, #6918: URL: https://github.com/apache/datafusion-comet/issues/6918
### What is the problem the feature request solves? When it writes nested columns, parquet-rs RLE-encodes the definition and repetition levels of every leaf. In `CometIcebergEndToEndBenchmark` (#6910), `RleEncoder::put` (9.4%) and `RleEncoder::flush_bit_packed_run` (2.8%) are 12% of all CPU for nested types, about 27% of the native writer. `LevelEncoder::put_with_observer` has a fast path for long runs of one level, but nested levels change every few values (lists of zero to nine elements, with nulls), so most of them take the per-value path through `put` and its 8-value buffer. ### Describe the potential solution Upstream in arrow-rs: an encoder that takes levels eight at a time, writing a bit-packed group directly when the group is mixed and starting or extending an RLE run when it is uniform. This would speed up every nested Parquet write Comet does, through both the native Iceberg writer and the native Parquet writer. ### Additional context parquet 59.3.0 (the version Comet uses): `RleEncoder::put` in `parquet/src/encodings/rle.rs` and `LevelEncoder::put_with_observer` in `parquet/src/encodings/levels.rs`. arrow-rs #7739 is about the decode cost of short RLE runs, not the encoding cost. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
