[
https://issues.apache.org/jira/browse/SPARK-60126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibo Dong updated SPARK-60126:
------------------------------
Labels: sql (was: )
> mode can return a less frequent value for MAP inputs
> ----------------------------------------------------
>
> Key: SPARK-60126
> URL: https://issues.apache.org/jira/browse/SPARK-60126
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark: 5.0.0-SNAPSHOT
> Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
> Java: OpenJDK 17.0.17
> OS: macOS 15.7.4
> Reporter: Yibo Dong
> Priority: Major
> Labels: sql
>
> h2. What happened
> Spark SQL's {{mode}} aggregate can return a less frequent {{MAP}} value even
> when another value is the unique most frequent value. The same query can
> produce different results, including incorrect ones.
> h2. How to reproduce
> Run the following SQL using {{{}spark-sql{}}}:
> {code:sql}
> SELECT g, mode(m) AS m
> FROM (
> SELECT g,
> CASE WHEN id <= 3
> THEN map(g, 'a', g + 100, 'b')
> ELSE map(g, 'x')
> END AS m
> FROM VALUES
> (0, 1), (0, 2), (0, 3), (0, 4),
> (1, 1), (1, 2), (1, 3), (1, 4),
> (2, 1), (2, 2), (2, 3), (2, 4),
> (3, 1), (3, 2), (3, 3), (3, 4),
> (4, 1), (4, 2), (4, 3), (4, 4),
> (5, 1), (5, 2), (5, 3), (5, 4),
> (6, 1), (6, 2), (6, 3), (6, 4),
> (7, 1), (7, 2), (7, 3), (7, 4)
> AS t(g, id)
> ) q
> GROUP BY g
> ORDER BY g;
> {code}
> Save the SQL as {{repro.sql}} and run:
> {code:bash}
> spark-sql --master 'local[2]'
> --conf spark.ui.enabled=false
> --conf spark.sql.shuffle.partitions=2
> -f repro.sql
> {code}
> h2. Expected result
> For each group, the two-entry map occurs three times, while the one-entry map
> occurs only once. Therefore, {{mode(m)}} should always return the two-entry
> map.
> {code}
> g | m
> --+----------------------
> 0 | {0 -> a, 100 -> b}
> 1 | {1 -> a, 101 -> b}
> 2 | {2 -> a, 102 -> b}
> 3 | {3 -> a, 103 -> b}
> 4 | {4 -> a, 104 -> b}
> 5 | {5 -> a, 105 -> b}
> 6 | {6 -> a, 106 -> b}
> 7 | {7 -> a, 107 -> b}
> {code}
> h2. Actual result
> The query can return a one-entry map that occurs only once, instead of the
> most frequent map. For example:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlg | m
> --+----------------------
> 0 | {0 -> x}
> 1 | {1 -> a, 101 -> b}
> 2 | {2 -> a, 102 -> b}
> 3 | {3 -> a, 103 -> b}
> 4 | {4 -> a, 104 -> b}
> 5 | {5 -> a, 105 -> b}
> 6 | {6 -> a, 106 -> b}
> 7 | {7 -> a, 107 -> b}
> {code}
> In this result, {{mode(m)}} selects {{0 -> x}} for group {{{}0{}}}, although
> that map occurs only once and the other map occurs three times.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
