[
https://issues.apache.org/jira/browse/ARROW-17719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17604954#comment-17604954
]
Philipp Moritz commented on ARROW-17719:
----------------------------------------
How do you feel about trying to unify the schema from the different files a bit
more / make schema inference a bit more powerful? This specific case of nulls
could be handled elegantly without raising an error then. We already have a
UnifySchemas function that might be usable for this. Concretely I'm proposing
to only handle the case where some files have a schema with type null for a
given field and others have a non-null type for that field (but all other types
are the same).
I realize that there is a fine line and it is possible to do too much
automation here but this seems like it won't cause problems. But let me know if
you see possible problems with this.
In any case we should also give a better error message because there will be
cases that this won't cover (e.g. nulls in a nested field).
> [Python] Improve error message when all values in a column are null in a
> parquet partition
> ------------------------------------------------------------------------------------------
>
> Key: ARROW-17719
> URL: https://issues.apache.org/jira/browse/ARROW-17719
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 9.0.0
> Reporter: Philipp Moritz
> Priority: Minor
> Fix For: 10.0.0
>
>
> There is a good bug report about this in
> [https://stackoverflow.com/a/70568419/10891801] and it still seems to be a
> problem.
> Basically the error message is pretty bad if all values in a given column of
> a parquet partition are null. We should either handle this case better or
> give a better error message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
