[
https://issues.apache.org/jira/browse/SPARK-59604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59604:
-----------------------------------
Labels: pull-request-available (was: )
> SimplifyExtractValueOps violates LAST_WIN semantics for duplicate map keys
> --------------------------------------------------------------------------
>
> Key: SPARK-59604
> URL: https://issues.apache.org/jira/browse/SPARK-59604
> Project: Spark
> Issue Type: Bug
> Components: Bug, Optimizer
> Affects Versions: 5.0.0
> Reporter: Yibo Dong
> Priority: Major
> Labels: pull-request-available
>
> h2. What happened
> {{SimplifyExtractValueOps}} changes the result of map subscripting when a map
> contains duplicate keys and {{spark.sql.mapKeyDedupPolicy}} is set to
> {{LAST_WIN}}.
> With the optimizer rule enabled, Spark returns the value associated with the
> first occurrence of the duplicate key. When only {{SimplifyExtractValueOps}}
> is excluded, Spark correctly returns the value associated with the last
> occurrence.
> h2. How to reproduce
> Run the following SQL:
> {code:sql}
> SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
> DROP TABLE IF EXISTS T;
> CREATE TABLE T USING PARQUET AS
> SELECT id
> FROM VALUES (1), (2) AS t(id);
> SELECT id, map(id, id, id, id + id)[id] AS value
> FROM T
> ORDER BY id;
> {code}
> Run it once with the default optimizer configuration:
> {code:bash}
> spark-sql
> --master 'local[2]'
> --conf spark.ui.enabled=false
> --conf spark.sql.adaptive.enabled=false
> --conf spark.sql.shuffle.partitions=1
> -f repro.sql
> {code}
> Then run the same query with only {{SimplifyExtractValueOps}} excluded:
> {code:bash}
> spark-sql
> --master 'local[2]'
> --conf spark.ui.enabled=false
> --conf spark.sql.adaptive.enabled=false
> --conf spark.sql.shuffle.partitions=1
> --conf
> spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.SimplifyExtractValueOps
>
> -f repro.sql
> {code}
> h2. Expected result
> With {{spark.sql.mapKeyDedupPolicy=LAST_WIN}}, the last value associated with
> a duplicate map key should be retained.
> For {{id = 1}}:
> {code}
> map(1, 1, 1, 2)[1] = 2
> {code}
> For {{id = 2}}:
> {code}
> map(2, 2, 2, 4)[2] = 4
> {code}
> Both optimizer configurations should therefore return:
> {code}
> 1 2
> 2 4
> {code}
> h2. Actual result
> With {{SimplifyExtractValueOps}} enabled, Spark returns:
> {code}
> 1 1
> 2 2
> {code}
> With only {{SimplifyExtractValueOps}} excluded, Spark returns:
> {code}
> 1 2
> 2 4
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
