[
https://issues.apache.org/jira/browse/SPARK-59326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Zdravic updated SPARK-59326:
---------------------------------
Description:
AsOfJoinMatchConditionTypesSuite unit-tests the MatchConditionTypes
operand-type predicates used by ASOF JOIN analysis, but omits the
non-orderable-type and struct field-type-mismatch cases. Those scenarios are
currently only exercised end-to-end (golden join-asof-errors.sql) or not at
all, with no direct unit assertion on the predicate.
Add:
- MAP operands are invalid (not orderable);
- ARRAY operands whose element type is non-orderable are invalid;
- STRUCT operands with a non-orderable field are invalid;
- positional STRUCT operands with the same field count but an incompatible
field-type pair are rejected (currently only field-count mismatch is covered).
- ARRAY operands whose element type contains an empty struct are invalid
(e.g. ARRAY<STRUCT<>>);
- ARRAY operands whose element structs have an incompatible field-type pair
are rejected.
Also add ResolveAsOfJoinSuite - direct unit tests for the fixed-point analyzer
rule ResolveAsOfJoin, which had no dedicated test (BIN BY has
ResolveBinBySuite). The rule materializes a MATCH_CONDITION into the executable
AsOfJoin fields and expands USING into equi-join predicates; previously
exercised only indirectly via SQL/golden/DataFrame suites.
Add:
- MATCH_CONDITION materializes into asOfCondition/orderExpression and
clears the match fields,
for all four operators (>=, >, <=, <);
- USING expands into an equi-join predicate wrapped in a Project (USING
columns tagged hidden);
- an explicit ON condition is preserved (no USING projection);
- INNER preserves right-side nullability, LEFT OUTER makes the right side
nullable;
- the three match-condition analyzer errors:
ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE,
_INVALID_EXPRESSION, _INVALID_TYPE;
- AsOfJoin survives the full Analyzer + CheckAnalysis.
Test-only; no production code changes. Complements the functional coverage in
AsOfJoinSQLSuite.
Umbrella: SPARK-59298
was:
AsOfJoinMatchConditionTypesSuite unit-tests the MatchConditionTypes
operand-type predicates used by ASOF JOIN analysis, but omits the
non-orderable-type and struct field-type-mismatch cases. Those scenarios are
currently only exercised end-to-end (golden join-asof-errors.sql) or not at
all, with no direct unit assertion on the predicate.
Add:
- MAP operands are invalid (not orderable);
- ARRAY operands whose element type is non-orderable are invalid;
- STRUCT operands with a non-orderable field are invalid;
- positional STRUCT operands with the same field count but an incompatible
field-type pair are rejected (currently only field-count mismatch is covered).
Test-only; no production code changes. Complements the functional coverage in
AsOfJoinSQLSuite.
Umbrella: SPARK-59298
Summary: Add ASOF JOIN analyzer resolution and match-condition type
unit tests (was: Add missing ASOF JOIN match-condition type unit tests)
> Add ASOF JOIN analyzer resolution and match-condition type unit tests
> ---------------------------------------------------------------------
>
> Key: SPARK-59326
> URL: https://issues.apache.org/jira/browse/SPARK-59326
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
>
> AsOfJoinMatchConditionTypesSuite unit-tests the MatchConditionTypes
> operand-type predicates used by ASOF JOIN analysis, but omits the
> non-orderable-type and struct field-type-mismatch cases. Those scenarios are
> currently only exercised end-to-end (golden join-asof-errors.sql) or not at
> all, with no direct unit assertion on the predicate.
> Add:
> - MAP operands are invalid (not orderable);
> - ARRAY operands whose element type is non-orderable are invalid;
> - STRUCT operands with a non-orderable field are invalid;
> - positional STRUCT operands with the same field count but an incompatible
> field-type pair are rejected (currently only field-count mismatch is covered).
> - ARRAY operands whose element type contains an empty struct are invalid
> (e.g. ARRAY<STRUCT<>>);
> - ARRAY operands whose element structs have an incompatible field-type
> pair are rejected.
> Also add ResolveAsOfJoinSuite - direct unit tests for the fixed-point
> analyzer rule ResolveAsOfJoin, which had no dedicated test (BIN BY has
> ResolveBinBySuite). The rule materializes a MATCH_CONDITION into the
> executable AsOfJoin fields and expands USING into equi-join predicates;
> previously exercised only indirectly via SQL/golden/DataFrame suites.
> Add:
> - MATCH_CONDITION materializes into asOfCondition/orderExpression and
> clears the match fields,
> for all four operators (>=, >, <=, <);
> - USING expands into an equi-join predicate wrapped in a Project (USING
> columns tagged hidden);
> - an explicit ON condition is preserved (no USING projection);
> - INNER preserves right-side nullability, LEFT OUTER makes the right side
> nullable;
> - the three match-condition analyzer errors:
> ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE,
> _INVALID_EXPRESSION, _INVALID_TYPE;
> - AsOfJoin survives the full Analyzer + CheckAnalysis.
>
> Test-only; no production code changes. Complements the functional coverage in
> AsOfJoinSQLSuite.
> Umbrella: SPARK-59298
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
