Haiyang Sun created SPARK-59709:
-----------------------------------
Summary: [UDF] Declared result nullability for external UDF workers
Key: SPARK-59709
URL: https://issues.apache.org/jira/browse/SPARK-59709
Project: Spark
Issue Type: Sub-task
Components: UDF
Affects Versions: 4.4.0
Reporter: Haiyang Sun
The language-agnostic external UDF expression carries declared result
nullability, but the current worker protocol does not enforce a non-null result
contract. Consequently, execution must conservatively expose all external UDF
results as nullable, matching PySpark’s behavior.
Add protocol and execution support for honoring declared non-nullability safely:
- Represent whether a worker can enforce non-null results.
- Use declared non-nullability only when the worker provides that guarantee.
- Reject a null result that violates the contract with a structured error.
- Propagate effective nullability consistently through logical and physical
planning.
- Preserve the current always-nullable behavior for Python workers without this
capability.
- Add tests covering downstream null checks and optimizer behavior, including
IS NULL.
Until this support exists, external UDF results remain conservatively nullable
to avoid incorrect null-based optimizations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
