jayshrivastava commented on issue #21207: URL: https://github.com/apache/datafusion/issues/21207#issuecomment-5026754906
👋🏽 I ran an expanded benchmark matrix here, comparing roughly the 4 implementations for dynamic filtering mentioned earlier: https://github.com/jayshrivastava/datafusion/blob/js/benchmark-4-dynamic-filtering-alternatives/benchmarks/reports/hash_join_dynamic_filter_expanded_matrix.md The main takeaway is that I do not think we have strong evidence for keeping the current partition-aware `CASE` expression as the default shape. In the benchmark, `CASE` does not participate in parquet row-group pruning, which matters the most for performance. The best-performing options were generally `global` (ie. `global_bounds AND global_membership`) and something very similar to option 2 you mentioned. Although, the non `CASE` parts of option 2 are by far the most important because they can be used for parquet pruning. The global filter implementation also gets an edge because it's simple. It would make dynamic filtering in distributed cases like DFD, ballista etc a bit easier to implement. Notably, trino basically just merges filters across partitions too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
