jayshrivastava commented on issue #23376:
URL: https://github.com/apache/datafusion/issues/23376#issuecomment-4970644515
ππ½ Prior to implementing this feature, would it make sense to implement
dynamic filtering **generally** first? For range partitioning, hash joins use
CASE expressions and I think we should do something similar here.
If you only implement dynamic filtering for copartitioned joins, you won't
support cases where the join is not co-partitioned, like below, which is likely
the vast majority of cases. However, if you work on the general case first,
where partitioning doesn't matter, then you will handle all cases, including
the co-partitioned case. IMO, it would be more valuable to make things work
generally and optimize for the co-partitioned case later.
Furthermore, `AggregateExec`, `HashJoinExec` (collect left), `SortExec` all
produce dynamic filters, but I are not covered by this issue.
I suggest creating 3 issues, one for `AggregateExec`, `SortExec`, and
`HashJoinExec`, and ensuring all cases of `HashJoinExec` work before starting
this co-partitioned optimization.
```
DynamicFilterPhysicalExpr:
ββββββββββββββββββββββββββ CASE 0 < a
<= 10:
β HashJoinExec: β CASE 11 <
a <= 20:
β mode=Partitioned β
....
ββββββββββββββββββββββββββ (1 case per
partition)
β² β²
β β β
β β β
ββββββββββββββββββ ββββββββββββββββββ β
β β β
β β β
β β β
ββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββ β
Pushed down to data source
β DataSourceExec: β β RepartitionExec: β β
βPartitioning=Range(a, 12) β βPartitioning=Range(a, 12) β β
ββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββ β
β² β
β β
β β
ββββββββββββββββββββββββββββ β
β DataSourceExec: β βΌ
β Partitioning=Unknown(10) β
ββββββββββββββββββββββββββββ
```
cc @peterxcli @gene-bordegaray
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
