alamb commented on code in PR #17293:
URL: https://github.com/apache/datafusion/pull/17293#discussion_r2294332793
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -409,64 +411,103 @@ impl FileOpener for ParquetOpener {
.with_row_groups(row_group_indexes)
.build()?;
- // Create a stateful stream that can check pruning after each batch
- let adapted = {
- use futures::stream;
- let schema_mapping = Some(schema_mapping);
- let file_pruner = file_pruner;
- let stream = stream.map_err(|e|
ArrowError::ExternalError(Box::new(e)));
- let files_ranges_pruned_statistics =
- file_metrics.files_ranges_pruned_statistics.clone();
-
- stream::try_unfold(
Review Comment:
the core idea is to move this try_unfold closure into a real structure --
all the logic is the same, but I think it is easier to understand in its own,
documented Stream struct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
