[
https://issues.apache.org/jira/browse/SPARK-59409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Zdravic updated SPARK-59409:
---------------------------------
Description:
RewriteAsOfJoinSuite (the unit test for the RewriteAsOfJoin optimizer rule)
previously contained only positive rewrite tests and left several parts of the
rule uncovered. This is a test-only change that closes those gaps and reduces
duplication.
Gaps closed:
* One (direction, tolerance, allowExactMatches) combination was missing:
nearest + allowExactMatches = false + no tolerance, whose as-of condition is
NOT(left = right).
* The "left outer" test actually built an Inner join (a duplicate of
"simple") and never tested LeftOuter; it now builds a real LeftOuter join,
isolating the one thing LeftOuter changes (omitting the IS NOT NULL filter).
* The rule's no-op guard
!conf.useSortMergeAsOfJoinOperator(requiresSortMergeAsOfJoin) had no coverage.
Added negative tests for both triggers: the requiresSortMergeAsOfJoin node
flag, and the sort-merge as-of config.
* The attribute remapping done by transformUpWithNewOutput was never
exercised (all tests put AsOfJoin at the plan root). Added a test with a
Project above the join.
Refactor:
* Introduced a shared expectedRewrite(filter, orderExpression, joinType)
helper so each test spells out only the parts that vary, removing ~15 lines of
duplicated plan-building per test. Mirrors the sibling
RewriteNearestByJoinSuite.
No production code changes; no user-facing change.
Umbrella: SPARK-59298
was:
RewriteAsOfJoinSuite (the unit test for the RewriteAsOfJoin optimizer rule)
previously contained only positive rewrite tests and left several parts of the
rule uncovered. This is a test-only change that closes those gaps and reduces
duplication.
Gaps closed:
* One (direction, tolerance, allowExactMatches) combination was missing:
nearest + allowExactMatches = false + no tolerance, whose as-of condition is
NOT(left = right).
* The "left outer" test actually built an Inner join (a duplicate of
"simple") and never tested LeftOuter; it now builds a real LeftOuter join,
isolating the one thing LeftOuter changes (omitting the IS NOT NULL filter).
* The rule's no-op guard
!conf.useSortMergeAsOfJoinOperator(requiresSortMergeAsOfJoin) had no coverage.
Added negative tests for both triggers: the requiresSortMergeAsOfJoin node
flag, and the sort-merge as-of config.
* The attribute remapping done by transformUpWithNewOutput was never
exercised (all tests put AsOfJoin at the plan root). Added a test with a
Project above the join.
Refactor:
* Introduced a shared expectedRewrite(filter, orderExpression, joinType)
helper so each test spells out only the parts that vary, removing ~15 lines of
duplicated plan-building per test. Mirrors the sibling
RewriteNearestByJoinSuite.
No production code changes; no user-facing change.
> Expand RewriteAsOfJoin optimizer rule test coverage
> ---------------------------------------------------
>
> Key: SPARK-59409
> URL: https://issues.apache.org/jira/browse/SPARK-59409
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Luka Zdravic
> Priority: Major
>
> RewriteAsOfJoinSuite (the unit test for the RewriteAsOfJoin optimizer rule)
> previously contained only positive rewrite tests and left several parts of
> the rule uncovered. This is a test-only change that closes those gaps and
> reduces duplication.
>
> Gaps closed:
> * One (direction, tolerance, allowExactMatches) combination was missing:
> nearest + allowExactMatches = false + no tolerance, whose as-of condition is
> NOT(left = right).
> * The "left outer" test actually built an Inner join (a duplicate of
> "simple") and never tested LeftOuter; it now builds a real LeftOuter join,
> isolating the one thing LeftOuter changes (omitting the IS NOT NULL filter).
> * The rule's no-op guard
> !conf.useSortMergeAsOfJoinOperator(requiresSortMergeAsOfJoin) had no
> coverage. Added negative tests for both triggers: the
> requiresSortMergeAsOfJoin node flag, and the sort-merge as-of config.
> * The attribute remapping done by transformUpWithNewOutput was never
> exercised (all tests put AsOfJoin at the plan root). Added a test with a
> Project above the join.
>
> Refactor:
> * Introduced a shared expectedRewrite(filter, orderExpression, joinType)
> helper so each test spells out only the parts that vary, removing ~15 lines
> of duplicated plan-building per test. Mirrors the sibling
> RewriteNearestByJoinSuite.
>
> No production code changes; no user-facing change.
> Umbrella: SPARK-59298
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
