jayshrivastava commented on issue #18856:
URL: https://github.com/apache/datafusion/issues/18856#issuecomment-5359395043
I'm interested in removing the execution-time check in `HashJoinExec` and
`AggregateExec` to see if a filter was pushed down. I think this should be done
at optimization time, not at execution time. This is the problem I'm running
into in datafusion-distributed:
```
worker 1
HashJoinExec (dynamic_filter_predicate)
NetworkShuffleExec
worker 2
DataSourceExec (dynamic_filter_predicate)
```
The hash join doesn't produce a filter because there's no filter used on
that worker. However, I would expect that the join to produce dynamic filters
because that decision should have been made during optimization time, before I
split up the plan.
I was wondering why this check even exists because
`handle_child_pushdown_result` should handle it. It turns out pruning
predicates return `No` indicating they weren't pushed down.
One alternative I considered was artificially pushing filters down to the
network nodes by doing a plan rewrite:
```
worker 1
HashJoinExec (dynamic_filter_predicate)
NetworkShuffleExec (dynamic_filter_predicate)
worker 2
DataSourceExec (dynamic_filter_predicate)
```
This feels like a smell and I'm not sure if it works in other projects. I
think other projects like ballista will run into this because they chop up
plans as well. IMO it's worth adding a new state to remove runtime filter
pushdown checks. I can try writing a small PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
