friendlymatthew commented on code in PR #20854:
URL: https://github.com/apache/datafusion/pull/20854#discussion_r2925996261
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -330,16 +345,17 @@ impl<'schema> PushdownChecker<'schema> {
fn check_struct_field_column(
&mut self,
column_name: &str,
+ field_path: Vec<String>,
) -> Option<TreeNodeRecursion> {
- let idx = match self.file_schema.index_of(column_name) {
- Ok(idx) => idx,
- Err(_) => {
- self.projected_columns = true;
- return Some(TreeNodeRecursion::Jump);
- }
+ let Ok(idx) = self.file_schema.index_of(column_name) else {
+ self.projected_columns = true;
+ return Some(TreeNodeRecursion::Jump);
Review Comment:
This can happen during schema evolution (a column that was added after the
file was written) or partition columns that are projected onto the scan but
don't exist physically on disk
In either case, we can't push the filter down because the decoder has no
data for this column
I'll add a comment explaining this or do it in a follow up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
