NikitaMatskevich commented on PR #5724: URL: https://github.com/apache/datafusion-comet/pull/5724#issuecomment-6413015518
A possible simplification after a dependency upgrade: parquet-rs 60.0.0 already includes opt-in dictionary-only Bloom suppression from [arrow-rs #10963](https://github.com/apache/arrow-rs/pull/10963), via [`set_bloom_filter_for_dictionary_encoded_chunks(false)`](https://docs.rs/parquet/60.0.0/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_bloom_filter_for_dictionary_encoded_chunks). Our separate [folding-disable PR, arrow-rs #11892](https://github.com/apache/arrow-rs/pull/11892), is still open; its target release is not confirmed. Once Comet adopts compatible dependencies containing both options, it could set them internally to retain Java-sized filters and omit dictionary-only filters, removing the reason for a folding-specific Comet opt-in. That would avoid the pruning tradeoff caused by folding. We would retain Java fallback for Bloom-enabled FLOAT/DOUBLE and BOOLEAN columns and resolve remaining physical-type differences, including the precision-1 decimal case, before broadening eligibility. Byte identity would apply to filters with matching physical values, allocation, and column-chunk contents; it would not guarantee identical whole-file or row-group layouts. This is a follow-up direction; the current PR retains its explicit opt-in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
