Yibo Dong created SPARK-59604:
---------------------------------
Summary: SimplifyExtractValueOps violates LAST_WIN semantics for
duplicate map keys
Key: SPARK-59604
URL: https://issues.apache.org/jira/browse/SPARK-59604
Project: Spark
Issue Type: Bug
Components: Bug, Optimizer
Affects Versions: 5.0.0
Reporter: Yibo Dong
h2. What happened
{{SimplifyExtractValueOps}} changes the result of map subscripting when a map
contains duplicate keys and {{spark.sql.mapKeyDedupPolicy}} is set to
{{LAST_WIN}}.
With the optimizer rule enabled, Spark returns the value associated with the
first occurrence of the duplicate key. When only {{SimplifyExtractValueOps}} is
excluded, Spark correctly returns the value associated with the last occurrence.
h2. How to reproduce
Run the following SQL:
{code:sql}
SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
DROP TABLE IF EXISTS T;
CREATE TABLE T USING PARQUET AS
SELECT id
FROM VALUES (1), (2) AS t(id);
SELECT id, map(id, id, id, id + id)[id] AS value
FROM T
ORDER BY id;
{code}
Run it once with the default optimizer configuration:
{code:bash}
spark-sql
--master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.adaptive.enabled=false
--conf spark.sql.shuffle.partitions=1
-f repro.sql
{code}
Then run the same query with only {{SimplifyExtractValueOps}} excluded:
{code:bash}
spark-sql
--master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.adaptive.enabled=false
--conf spark.sql.shuffle.partitions=1
--conf
spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.SimplifyExtractValueOps
-f repro.sql
{code}
h2. Expected result
With {{spark.sql.mapKeyDedupPolicy=LAST_WIN}}, the last value associated with a
duplicate map key should be retained.
For {{id = 1}}:
{code}
map(1, 1, 1, 2)[1] = 2
{code}
For {{id = 2}}:
{code}
map(2, 2, 2, 4)[2] = 4
{code}
Both optimizer configurations should therefore return:
{code}
1 2
2 4
{code}
h2. Actual result
With {{SimplifyExtractValueOps}} enabled, Spark returns:
{code}
1 1
2 2
{code}
With only {{SimplifyExtractValueOps}} excluded, Spark returns:
{code}
1 2
2 4
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
