[
https://issues.apache.org/jira/browse/SPARK-59681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59681:
-----------------------------------
Labels: pull-request-available (was: )
> Guaranteed shuffle reuse for materialized/not inlined CTE
> ---------------------------------------------------------
>
> Key: SPARK-59681
> URL: https://issues.apache.org/jira/browse/SPARK-59681
> Project: Spark
> Issue Type: Task
> Components: Optimizer
> Affects Versions: 4.4.0
> Reporter: Avery Qi
> Priority: Major
> Labels: pull-request-available
>
> A CTE referenced multiple times can be either inlined (recomputed per
> reference) or materialized once. For a *non-deterministic* body (e.g.
> {{{}rand(){}}}, {{{}monotonically_increasing_id(){}}}, non-deterministic
> UDFs), recomputation is not just wasteful but *semantically wrong* — each
> reference would observe different rows.
>
> Currently spark already has CTE which is marked as `materialized` or
> `forceSkipInline` but the CTE is being changed into Repartition over
> subplans. Repartition leads to a normal shuffle which can be changed or
> removed in the physical planning stages.
>
> This task provide a guaranteed shuffle reuse for CTE which is marked as
> `forceSkipInline`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
