[
https://issues.apache.org/jira/browse/ARROW-17719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605791#comment-17605791
]
Weston Pace commented on ARROW-17719:
-------------------------------------
The performance impact is pretty small when you have only a few files.
However, if you have thousands of files, this can significantly impact
performance (for these larger datasets even the cost of calling {{stat}} on
each file to get the file info becomes significant). I'm not strongly against
changing the default but maybe it would be better to just do a better job of
explaining when the flag is needed so that future users find it more quickly?
I think having all null in a column in earlier files is not a common enough
thing to justify hurting performance.
> [Python] Improve error message when all values in a column are null in a
> parquet partition
> ------------------------------------------------------------------------------------------
>
> Key: ARROW-17719
> URL: https://issues.apache.org/jira/browse/ARROW-17719
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 9.0.0
> Reporter: Philipp Moritz
> Priority: Minor
> Fix For: 10.0.0
>
>
> There is a good bug report about this in
> [https://stackoverflow.com/a/70568419/10891801] and it still seems to be a
> problem.
> Basically the error message is pretty bad if all values in a given column of
> a parquet partition are null. We should either handle this case better or
> give a better error message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
