Joel Robin created SPARK-59097:
----------------------------------
Summary: UnsafeProjection codegen throws NullPointerException on a
null in a nested non-nullable field, where the interpreted path writes null
Key: SPARK-59097
URL: https://issues.apache.org/jira/browse/SPARK-59097
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0, 3.5.2, 5.0.0
Reporter: Joel Robin
A struct, array or map value that is declared non-nullable but is actually null
at runtime gets handled two different ways depending on which UnsafeProjection
you end up with. The interpreted one writes null and carries on. The generated
one dereferences the null and throws. Both are implementations of the same
thing, so which one you get should not decide whether the query survives.
It surfaces as a task failure like this:
{code}
java.lang.NullPointerException: Cannot invoke
"org.apache.spark.sql.catalyst.util.ArrayData.numElements()" because
"<local10>" is null
at ...GeneratedClass$SpecificUnsafeProjection.writeFields_1_3$(Unknown Source)
at ...GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at ...GeneratedClass$GeneratedIteratorForCodegenStage31.processNext(Unknown
Source)
at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:154)
{code}
h3. Why codegen throws
writeExpressionsToBuffer drops the null check whenever the declared nullability
says the value cannot be null (GenerateUnsafeProjection.scala:156):
{code:scala}
if (!nullable || input.isNull == FalseLiteral) {
// The value is statically known to be non-null, so skip the null check and
the
// (dead) setNull branch and just write the value.
{code}
Array elements get the same treatment when containsNull is false (:248-249).
writeArrayToBuffer then does the dereference (:257):
{code:java}
final ArrayData tmpInput = <input>;
if (tmpInput instanceof UnsafeArrayData) { ... }
else { final int numElements = tmpInput.numElements(); // throws here
{code}
tmpInput is the <localN> named in the NPE message.
InterpretedUnsafeProjection.generateFieldWriter does the opposite, and the
comment is explicit about it (InterpretedUnsafeProjection.scala:251):
{code:scala}
// Always wrap the writer with a null safe version.
{code}
It does take nullable as a parameter, but only to work out the child writer's
nullability. It
never consults it when deciding whether to null-check. So the interpreted path
tolerates an
inaccurate declaration and codegen does not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
