Luka Zdravic created SPARK-59749:
------------------------------------
Summary: ASOF JOIN MATCH_CONDITION rejects empty STRUCT operands
accepted by the comparison operator
Key: SPARK-59749
URL: https://issues.apache.org/jira/browse/SPARK-59749
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Luka Zdravic
Inside a MATCH_CONDITION, comparing two empty structs (STRUCT<>) fails with
ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE. The same comparison outside ASOF works:
two empty structs compare as equal. The error says the operands must be
orderable, but STRUCT<> is orderable. ASOF adds its own check that rejects any
type that contains an empty struct.
SET spark.sql.join.asofJoin.enabled = true;
-- fails:
SELECT * FROM VALUES (named_struct()) AS l(s)
ASOF JOIN VALUES (named_struct()) AS r(s)
MATCH_CONDITION (l.s >= r.s); -- ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE:
"STRUCT<>" and "STRUCT<>"
-- works:
SELECT named_struct() >= named_struct(); -- true
The same check also rejects nested empty structs, for example STRUCT<x:
STRUCT<>> and ARRAY<STRUCT<>>.
Expected: MATCH_CONDITION accepts empty struct operands, nested or not, same as
the comparison operator. An empty struct paired with a non-empty struct stays
rejected.
The check does not depend on ANSI mode.
Impact: an as-of condition that is a legal comparison cannot be written.
Umbrella: SPARK-59526
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
