adriangb commented on code in PR #15769:
URL: https://github.com/apache/datafusion/pull/15769#discussion_r2074193501
##########
datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt:
##########
@@ -221,5 +226,23 @@ physical_plan DataSourceExec: file_groups={2 groups:
[[WORKSPACE_ROOT/datafusion
query TT
EXPLAIN select * from t_pushdown where val != 'c';
----
-logical_plan TableScan: t_pushdown projection=[val, part],
full_filters=[t_pushdown.val != Utf8("c")]
+logical_plan
+01)Filter: t_pushdown.val != Utf8("c")
+02)--TableScan: t_pushdown projection=[val, part],
partial_filters=[t_pushdown.val != Utf8("c")]
physical_plan DataSourceExec: file_groups={3 groups:
[[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_part_test/part=a/file.parquet],
[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_part_test/part=b/file.parquet],
[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_part_test/part=c/file.parquet]]},
projection=[val, part], file_type=parquet, predicate=val@0 != c,
pruning_predicate=val_null_count@2 != row_count@3 AND (val_min@0 != c OR c !=
val_max@1), required_guarantees=[val not in (c)]
+
+# If we have a mix of filters:
+# - The partition filters get evaluated during planning
+# - The mixed filters end up in a FilterExec
+# - The file filters get pushed down into the scan
+query TT
+EXPLAIN select * from t_pushdown where val != 'd' AND val != 'c' AND part =
'a' AND part != val;
+----
+logical_plan
+01)Filter: t_pushdown.val != Utf8("d") AND t_pushdown.val != Utf8("c") AND
t_pushdown.val != t_pushdown.part
+02)--TableScan: t_pushdown projection=[val, part],
full_filters=[t_pushdown.part = Utf8("a")], partial_filters=[t_pushdown.val !=
Utf8("d"), t_pushdown.val != Utf8("c"), t_pushdown.val != t_pushdown.part]
+physical_plan
+01)CoalesceBatchesExec: target_batch_size=8192
+02)--FilterExec: val@0 != part@1
+03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+04)------DataSourceExec: file_groups={1 group:
[[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_part_test/part=a/file.parquet]]},
projection=[val, part], file_type=parquet, predicate=val@0 != d AND val@0 !=
c, pruning_predicate=val_null_count@2 != row_count@3 AND (val_min@0 != d OR d
!= val_max@1) AND val_null_count@2 != row_count@3 AND (val_min@0 != c OR c !=
val_max@1), required_guarantees=[val not in (c, d)]
Review Comment:
Replaces deleted test
##########
datafusion/core/tests/sql/path_partition.rs:
##########
@@ -57,55 +53,6 @@ use object_store::{
use object_store::{Attributes, MultipartUpload, PutMultipartOpts, PutPayload};
use url::Url;
-#[tokio::test]
-async fn parquet_partition_pruning_filter() -> Result<()> {
Review Comment:
It was relying on `ListingTable` to do the pushdown without the optimizer
pass.
I felt that it was better suited for an `.slt` test that is fully wired up,
so I essentially moved it:
https://github.com/apache/datafusion/pull/15769/files#r2074193501
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
