github-actions[bot] closed pull request #13054: JoinOptimization: Add build
side pushdown to probe side
URL: https://github.com/apache/datafusion/pull/13054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
github-actions[bot] commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2591473073
Thank you for your contribution. Unfortunately, this pull request is stale
because it has been open 60 days with no activity. Please remove the stale
label or comment or
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2477812838
https://github.com/user-attachments/assets/0d4015d8-435f-4f57-bb1f-fb17a574dd48";>
--
This is an automated message from the Apache Git Service.
To respond to the message, plea
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2475214828
I repushed with an adaptive way to generate dynamic filter(when unique value
exceed a threshold, generate range filter, otherwise inlist filter), and due to
https://github.com/apac
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2463427276
I decided not to create PhysicalExpr for it since the only benefit of
creating extra PhysicalExpr is try to pushdown it to ParquetScan, but currently
the adaptive feature of pushdo
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1833251081
##
datafusion/physical-plan/src/joins/hash_join.rs:
##
@@ -1406,12 +1403,24 @@ impl HashJoinStream {
self.hashes_buffer.resize(batch.num_rows(),
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1833251081
##
datafusion/physical-plan/src/joins/hash_join.rs:
##
@@ -1406,12 +1403,24 @@ impl HashJoinStream {
self.hashes_buffer.resize(batch.num_rows(),
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1833251081
##
datafusion/physical-plan/src/joins/hash_join.rs:
##
@@ -1406,12 +1403,24 @@ impl HashJoinStream {
self.hashes_buffer.resize(batch.num_rows(),
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1832226325
##
datafusion/physical-plan/src/joins/hash_join.rs:
##
@@ -1406,12 +1403,24 @@ impl HashJoinStream {
self.hashes_buffer.resize(batch.num_rows(),
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2461159231
Another weird thing I found after rebasing main is that the ParquetExec time
seems to be wrong
without optimization
https://github.com/user-attachments/assets/6207c16e-69e2-45
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1831785310
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1831783035
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1831707315
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1825101656
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1824670958
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2434908233
> > How would you display them in sources? The dynamic filter will only be
added during execution, so it will only be available through e.g. ParquetExec
after loading the build sid
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2436827458
I think it is ready now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2436724056
> > How would you display them in sources? The dynamic filter will only be
added during execution, so it will only be available through e.g. ParquetExec
after loading the build sid
berkaysynnada commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2434890005
> How would you display them in sources? The dynamic filter will only be
added during execution, so it will only be available through e.g. ParquetExec
after loading the build s
Dandandan commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2434605384
> Could you update the display of sources to include these pushed-down
filters? I believe it would be more clear if we could see them clearly in the
output (I'm also curious to see
berkaysynnada commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2434455783
Could you update the display of sources to include these pushed-down
filters? I believe it would be more clear if we could see them clearly in the
output (I'm also curious to s
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1813994423
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -243,6 +231,65 @@ impl DynamicFilterInfo {
}
}
+macro_rules! process_min_max {
+($ARR
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1813300852
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811979076
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811959953
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -243,6 +231,65 @@ impl DynamicFilterInfo {
}
}
+macro_rules! process_min_max {
+($ARR
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811912253
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811912253
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811568285
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -243,6 +231,65 @@ impl DynamicFilterInfo {
}
}
+macro_rules! process_min_max {
+($ARR
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r181139
##
datafusion/core/src/physical_optimizer/join_filter_pushdown.rs:
##
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or mor
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811372356
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811372047
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811370714
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811370182
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811367135
##
datafusion/core/src/datasource/physical_plan/parquet/mod.rs:
##
@@ -711,10 +724,15 @@ impl DisplayAs for ParquetExec {
)
Lordworms commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r181139
##
datafusion/core/src/physical_optimizer/join_filter_pushdown.rs:
##
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or mor
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1811141278
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810705503
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810153790
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810178550
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810178550
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810178550
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810153790
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810153790
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810149645
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810118665
##
datafusion/core/src/physical_optimizer/join_filter_pushdown.rs:
##
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or mor
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810141838
##
datafusion/physical-plan/src/joins/dynamic_filters.rs:
##
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contrib
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810119820
##
datafusion/core/src/physical_optimizer/join_filter_pushdown.rs:
##
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or mor
Dandandan commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810116130
##
datafusion/core/src/datasource/physical_plan/parquet/mod.rs:
##
@@ -711,10 +724,15 @@ impl DisplayAs for ParquetExec {
)
simonvandel commented on code in PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#discussion_r1810049282
##
datafusion/core/src/physical_optimizer/join_filter_pushdown.rs:
##
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or m
Dandandan commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2428351362
Really nice @Lordworms , will have a good look later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2428075067
@Dandandan Sorry for the late response, I have reinvested it and format the
code. Really appreciate your suggestion. Thanks a lot.
--
This is an automated message from the Apache
Lordworms commented on PR #13054:
URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2428074119
For query like
```
select count(*) from part join lineitem on l_partkey=p_partkey where
p_partkey > 10;
```
and with data scale like
