[
https://issues.apache.org/jira/browse/SPARK-59529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Zdravic updated SPARK-59529:
---------------------------------
Description:
MATCH_CONDITION should require each operand to reference exactly one join input
(one side left, the other right). Spark enforces this for a wrong-side
reference but not for a no-reference operand: a literal, or a call like
current_date(), is accepted and pinned to its syntactic side, so the query
builds a real ASOF join with a constant operand.
Confirmed on commit 083d824, ANSI off and on:
-- accepted, builds AsOfJoin (a >= 5), Inner:
SELECT * FROM (VALUES (1),(6)) l(a) ASOF JOIN (VALUES (10)) r(b)
MATCH_CONDITION (l.a >= 5);
-- accepted:
SELECT * FROM (VALUES (DATE'2024-01-01')) l(a) ASOF JOIN (VALUES
(DATE'2024-01-02')) r(b)
MATCH_CONDITION (l.a >= current_date());
-- already rejected today with ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE (both
operands on one side):
SELECT * FROM (VALUES (1,2)) l(a,c) ASOF JOIN (VALUES (10)) r(b)
MATCH_CONDITION (l.a >= l.c);
Expected: reject a MATCH_CONDITION operand that references no join input,
reusing the existing ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE error. The
check already exists for wrong-side references; extend it to the no-reference
case.
Impact: a query that reads like an as-of join builds with a constant operand
and no defined nearest-match. Low priority.
Umbrella: SPARK-59526
was:
MATCH_CONDITION accepts an operand that references no join input, such as a
literal or current_date(). Spark assigns the constant to its syntactic side and
builds the join. The as-of match compares one left column to one right column
to find the nearest row, so an operand that references neither input has no
well-defined meaning.
SELECT * FROM t ASOF JOIN u
MATCH_CONDITION (t.ts >= current_date()); -- accepted today
Expected: reject a MATCH_CONDITION operand that does not reference exactly one
join input.
Impact: a query that reads like an as-of join runs with no defined
nearest-match.
Umbrella: SPARK-59526
> ASOF JOIN MATCH_CONDITION accepts a constant operand referencing no join input
> ------------------------------------------------------------------------------
>
> Key: SPARK-59529
> URL: https://issues.apache.org/jira/browse/SPARK-59529
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Luka Zdravic
> Priority: Minor
>
> MATCH_CONDITION should require each operand to reference exactly one join
> input (one side left, the other right). Spark enforces this for a wrong-side
> reference but not for a no-reference operand: a literal, or a call like
> current_date(), is accepted and pinned to its syntactic side, so the query
> builds a real ASOF join with a constant operand.
> Confirmed on commit 083d824, ANSI off and on:
> -- accepted, builds AsOfJoin (a >= 5), Inner:
> SELECT * FROM (VALUES (1),(6)) l(a) ASOF JOIN (VALUES (10)) r(b)
> MATCH_CONDITION (l.a >= 5);
> -- accepted:
> SELECT * FROM (VALUES (DATE'2024-01-01')) l(a) ASOF JOIN (VALUES
> (DATE'2024-01-02')) r(b)
> MATCH_CONDITION (l.a >= current_date());
> -- already rejected today with ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE
> (both operands on one side):
> SELECT * FROM (VALUES (1,2)) l(a,c) ASOF JOIN (VALUES (10)) r(b)
> MATCH_CONDITION (l.a >= l.c);
> Expected: reject a MATCH_CONDITION operand that references no join input,
> reusing the existing ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE error. The
> check already exists for wrong-side references; extend it to the no-reference
> case.
> Impact: a query that reads like an as-of join builds with a constant operand
> and no defined nearest-match. Low priority.
> Umbrella: SPARK-59526
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
