Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/1889#issuecomment-51918543
@ueshin If we are changing the default value of `ArrayType.containsNull` to
`true`, we may also need to add logic to set that value based on data sources.
In
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetConverter.scala#L78,
currently, there is no pattern for `ArrayType.containsNull=true`. If we do not
add logic to set `ArrayType.containsNull` or support null values in arrays in
our Parquet support (I am not sure if Parquet can do this, but seems people are
working on it for Hive https://issues.apache.org/jira/browse/HIVE-6994), seems
some existing working cases will break.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
