tustvold commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603007745
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
To emphasise the point I made when this API was originally proposed, you
need more than just the ParquetStatistics in order to correctly interpret the
data. You need at least the FileMetadata to get the
https://docs.rs/parquet/latest/parquet/file/metadata/struct.FileMetaData.html#method.column_order
in order to be able to even interpret what the statistics mean for a given
column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
