alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603344193
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
In terms of "column order" I think we initially should do what DataFusion
currently does with ColumnOrder (which is ignore it) and file a ticket to
handle it longer term
Including the parquet schema is a good idea. I think this will become more
obvious as we begin writing these tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
