[
https://issues.apache.org/jira/browse/SPARK-59598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59598:
-----------------------------------
Labels: pull-request-available (was: )
> GetMapValue and ElementAt must not match a null map key
> -------------------------------------------------------
>
> Key: SPARK-59598
> URL: https://issues.apache.org/jira/browse/SPARK-59598
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Qing Fu
> Priority: Major
> Labels: pull-request-available
>
> {{GetMapValue}} ({{m[k]}}) and {{ElementAt}} ({{element_at(m, k)}}) share
> their key lookup in {{GetMapValueUtil}}. When the map's key array contains a
> null, the lookup matches that null key against the key type's zero value and
> returns its value instead of NULL.
> For a map {{{}{null: 10, 1: 20}{}}}:
> {code}
> m[0] ==> 10 -- should be NULL
> m[1] ==> 20 -- correct
> {code}
> Two independent mechanisms cause this, so it reproduces with codegen both on
> and off:
> 1. Interpreted linear scan. {{ordering.equiv(keys.get(i, keyType), ordinal)}}
> compares using the key type's natural ordering. {{keys.get}} is null-aware
> and correctly returns null for a null slot, but for a primitive key type the
> ordering unboxes its arguments, and Scala's {{BoxesRunTime.unboxToInt(null)}}
> is 0. So {{equiv(null, 0)}} is true.
> 2. Generated code. The candidate key is read with a primitive getter
> ({{CodeGenerator.getValue}} emits {{keys.getInt(i)}}), which is not
> null-aware and returns 0 for a null slot. This affects the generated linear
> scan, and also the hash probe added by SPARK-55959: {{buildHashBuckets}}
> hashes the unboxed null into the same bucket as 0, so the probe finds the
> null key and then compares 0 == 0.
> Maps with null keys are reachable. {{ArrayBasedMapBuilder}} rejects them, but
> the file-format readers construct {{ArrayBasedMapData}} directly and do not.
> {{ParquetRowConverter}} says so in the tree today:
> {code:scala}
> override def end(): Unit = {
> // The parquet map may contains null or duplicated map keys. When it
> happens, the behavior is
> // undefined.
> // TODO (SPARK-26174): disallow it with a config.
> {code}
> {{OrcDeserializer}} and {{AvroDeserializer}} likewise build the key array
> without a runtime null check, and the Hive {{MapObjectInspector}} unwrapper
> in {{HiveInspectors}} applies the key unwrapper with no null check at all. So
> a null key can reach the lookup from Parquet, ORC, Avro or Hive data.
> The lookup key itself is never null (both expressions are null-safe on the
> ordinal), so a null map key can never be the key being looked up. Skipping
> null keys makes the result well-defined and consistent across all paths, and
> changes nothing for maps without null keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
