kosiew commented on code in PR #24770:
URL: https://github.com/apache/datafusion/pull/24770#discussion_r3942870045
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -794,10 +799,29 @@ impl ParquetMorselizer {
// Note that if there are statistics for partition columns there will
be overlap,
// but since we use a HashMap, we'll just overwrite the partition
values with the
// constant values from statistics (which should be the same).
- literal_columns.extend(constant_columns_from_stats(
+ let stats_constants = constant_columns_from_stats(
Review Comment:
I think there is still a schema-evolution hole here.
`DFParquetMetadata::statistics_from_parquet_metadata` gives a logical column
that is absent from the physical Parquet schema `null_count = Exact(num_rows)`.
`constant_value_from_stats` then turns that into a NULL entry in
`stats_constants`.
So for a file that physically contains only `a`, collected stats can still
make missing `b` look like a stats-derived NULL constant. With `a = 1 AND b =
2`, `stats_alone_unsatisfiable` can simplify the whole predicate to NULL, set
`stats_prove_unsatisfiable`, and skip the row group through the statistics path.
That seems to bring back the missing-column row-group skip that this change
is trying to avoid. The new mixed regression currently leaves `b` statistics
unknown, so it does not reproduce what real collected schema-evolution
statistics provide.
Could we preserve whether an all-NULL constant came from a physically absent
column, or defer this attribution until the physical schema is known? I would
also update the regression so `b.null_count = Exact(3)`, or better, use real
collected statistics for the missing-column case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
