comphead commented on PR #27278: URL: https://github.com/apache/datafusion/pull/27278#issuecomment-6801727359
Thanks @andygrove, @vinycalves, @mbutrovich, @sunchao and @viirya. 8012102c4 takes the scope suggestion: the join now filters at freeze only. Filtering while pairing is gone, and with it the run threshold, `FILTERED_CHUNK_KEY`, the `filtered` flag, `Cow` and the spill fallback. `materializing_stream.rs` is now +371/-91 against `main`, down from +667/-104. These are local medians of 3 alternating rounds on an Apple M5 Max, against 9d916ab47 (`main` plus the bench commit): | Case | `main` | with filtering while pairing | at freeze only | |---|---:|---:|---:| | `inner_1to1000_selective_filtered` | 13.56 ms | 2.28 ms | 3.18 ms | | 1:256 `key, data`, 90% kept | 5.71 ms | 8.68 ms | 5.67 ms | | 1:512 `key, data`, 90% kept | 5.75 ms | 7.69 ms | 5.74 ms | Filtering while pairing was up to 1.4x faster for selective filters on runs of 512 rows or more. It was also up to 1.5x slower than `main` when the filter keeps most pairs and the schema is narrow. I haven't rerun q72 in Comet with filtering at freeze only yet, and will post that number here. If q72 needs filtering while pairing, it can come back as its own PR. The inline replies cover the rest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
