adriangb commented on code in PR #18172:
URL: https://github.com/apache/datafusion/pull/18172#discussion_r2446745073
##########
datafusion/datasource-parquet/src/source.rs:
##########
@@ -720,7 +721,9 @@ impl FileSource for ParquetSource {
let filters: Vec<PushedDownPredicate> = filters
.into_iter()
.map(|filter| {
- if can_expr_be_pushed_down_with_schemas(&filter, &file_schema)
{
+ if is_dynamic_physical_expr(&filter)
+ || can_expr_be_pushed_down_with_schemas(&filter,
&file_schema)
Review Comment:
> For a filter found in a query, if it contains a partition column, it would
be partition pruned by table provider.
> The only case where this function would receive a filter expr containing
partition column, is for dynamic filter
I don't think this is right. Consider `file_data_column < partition_column`
as a filter. That can only be pruned once we have the file data, despite
referring to a partition column. This actually used to be a bug in DataFusion,
fixed in https://github.com/apache/datafusion/pull/15057 I believe. Now
`ParquetOpener` can evaluate filters that reference partition columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
