andygrove opened a new issue, #6921: URL: https://github.com/apache/datafusion-comet/issues/6921
### What is the problem the feature request solves? The native Iceberg scan is 14% of all CPU for nested types in `CometIcebergEndToEndBenchmark` (#6910). Most of it is parquet-rs rebuilding nested arrays: - `build_filtered_validity_bitmap`: 2.6% - `ListArrayReader::consume_batch`: 2.5% - `read_rep_levels`: 1.9% - null padding (`pad_nulls`): 1.1% - dictionary string decoding (`extend_from_dictionary`): 1% - copies The scan is already a large win: Spark's iceberg-java row reader spends 2.9 CPU-seconds per iteration on the same data against Comet's 0.8. ### Describe the potential solution Upstream in arrow-rs: vectorize rebuilding list offsets from repetition levels and turning definition levels into validity bitmaps. #9848 (selective null padding for list child readers, already in parquet 59.3.0) and the `ListArray` benchmarks from #9846 are a starting point. ### Additional context parquet 59.3.0 (the version Comet uses): `parquet/src/arrow/array_reader/list_array.rs` and `parquet/src/arrow/record_reader/definition_levels.rs`. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
