Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1889#discussion_r16094479
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
---
@@ -372,7 +372,7 @@ object MapType {
* The `valueContainsNull` is true.
*/
def apply(keyType: DataType, valueType: DataType): MapType =
- MapType(keyType: DataType, valueType: DataType, true)
+ MapType(keyType: DataType, valueType: DataType, false)
--- End diff --
Hmm, that actually makes me think that we should change ArrayTypes default
value as well. Here is my reason: Marking something as not nullable is purely
an optimization as adding null checking logic will never cause the answer to be
incorrect. If the user wants that optimization great, but it seems dangerous
to assume that we can apply it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
