adriangb commented on code in PR #19111:
URL: https://github.com/apache/datafusion/pull/19111#discussion_r2593336195
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -139,8 +140,26 @@ impl FileOpener for ParquetOpener {
let batch_size = self.batch_size;
- let projection = Arc::clone(&self.projection);
+ // Get the projection, replacing partition columns with literals
+ let partition_values = std::mem::take(&mut
partitioned_file.partition_values);
+ let partition_values = partition_values
+ .into_iter()
+ .zip(self.partition_fields.iter())
+ .map(|(value, field)| (field.name().as_str(), value))
+ .collect::<HashMap<_, _>>();
+ let mut projection = self.projection.clone();
+ projection = projection.try_map_exprs(|expr| {
+ replace_columns_with_literals(Arc::clone(&expr), &partition_values)
+ })?;
Review Comment:
I don't particularly love this bit of code. I didn't want to make
`replace_columns_with_literals` depend on a `FieldRef` so that it can be used
in other contexts. Building a HashMap seems like the logical thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
