Qing Fu created SPARK-59598:
-------------------------------
Summary: GetMapValue and ElementAt must not match a null map key
Key: SPARK-59598
URL: https://issues.apache.org/jira/browse/SPARK-59598
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Qing Fu
{{GetMapValue}} ({{m[k]}}) and {{ElementAt}} ({{element_at(m, k)}}) share their
key lookup in {{GetMapValueUtil}}. When the map's key array contains a null,
the lookup matches that null key against the key type's zero value and returns
its value instead of NULL.
For a map {{{}{null: 10, 1: 20}{}}}:
{code}
m[0] ==> 10 -- should be NULL
m[1] ==> 20 -- correct
{code}
Two independent mechanisms cause this, so it reproduces with codegen both on
and off:
1. Interpreted linear scan. {{ordering.equiv(keys.get(i, keyType), ordinal)}}
compares using the key type's natural ordering. {{keys.get}} is null-aware and
correctly returns null for a null slot, but for a primitive key type the
ordering unboxes its arguments, and Scala's {{BoxesRunTime.unboxToInt(null)}}
is 0. So {{equiv(null, 0)}} is true.
2. Generated code. The candidate key is read with a primitive getter
({{CodeGenerator.getValue}} emits {{keys.getInt(i)}}), which is not null-aware
and returns 0 for a null slot. This affects the generated linear scan, and also
the hash probe added by SPARK-55959: {{buildHashBuckets}} hashes the unboxed
null into the same bucket as 0, so the probe finds the null key and then
compares 0 == 0.
Maps with null keys are reachable. {{ArrayBasedMapBuilder}} rejects them, but
the file-format readers construct {{ArrayBasedMapData}} directly and do not.
{{ParquetRowConverter}} says so in the tree today:
{code:scala}
override def end(): Unit = {
// The parquet map may contains null or duplicated map keys. When it happens,
the behavior is
// undefined.
// TODO (SPARK-26174): disallow it with a config.
{code}
{{OrcDeserializer}} and {{AvroDeserializer}} likewise build the key array
without a runtime null check, and the Hive {{MapObjectInspector}} unwrapper in
{{HiveInspectors}} applies the key unwrapper with no null check at all. So a
null key can reach the lookup from Parquet, ORC, Avro or Hive data.
The lookup key itself is never null (both expressions are null-safe on the
ordinal), so a null map key can never be the key being looked up. Skipping null
keys makes the result well-defined and consistent across all paths, and changes
nothing for maps without null keys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
