QinXi-ai commented on PR #25159: URL: https://github.com/apache/datafusion/pull/25159#issuecomment-5621723633
I am opening this as a draft to get feedback on the design direction before extending the implementation. The main questions I would appreciate feedback on are: 1. Is a private, probe-side selection exchange owned by HashJoinExec an acceptable first step for #15382, or should this wait for a reusable selection representation across operators? 2. Is the initial scope useful: opt-in partitioned INNER joins, direct matching hash repartition, and unlimited memory pools? In particular, the original-batch retention increases peak reservations in the low-match Utf8 case even though the measured process working set decreases. 3. What benchmark and validation gates would you want before considering this ready? The Windows operator results show substantial gains for wide, low-match inputs, regressions for some narrow inputs, and noisy default-off comparisons. Full workspace/SLT completion and Linux CI are still outstanding as detailed in the description. The intent is to evaluate this limited design, not to close the broader issue or enable it by default. I would be happy to split out the benchmark or adjust the integration boundary based on maintainer feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
