Yibo Dong created SPARK-60126:
---------------------------------
Summary: mode can return a less frequent value for MAP inputs
Key: SPARK-60126
URL: https://issues.apache.org/jira/browse/SPARK-60126
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark: 5.0.0-SNAPSHOT
Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Reporter: Yibo Dong
h2. What happened
Spark SQL's {{mode}} aggregate can return a less frequent {{MAP}} value even
when another value is the unique most frequent value. The same query can
produce different results, including incorrect ones.
h2. How to reproduce
Run the following SQL using {{spark-sql}}:
{code:sql}
SELECT g, mode(m) AS m
FROM (
SELECT g,
CASE WHEN id <= 3
THEN map(g, 'a', g + 100, 'b')
ELSE map(g, 'x')
END AS m
FROM VALUES
(0, 1), (0, 2), (0, 3), (0, 4),
(1, 1), (1, 2), (1, 3), (1, 4),
(2, 1), (2, 2), (2, 3), (2, 4),
(3, 1), (3, 2), (3, 3), (3, 4),
(4, 1), (4, 2), (4, 3), (4, 4),
(5, 1), (5, 2), (5, 3), (5, 4),
(6, 1), (6, 2), (6, 3), (6, 4),
(7, 1), (7, 2), (7, 3), (7, 4)
AS t(g, id)
) q
GROUP BY g
ORDER BY g;
{code}
Save the SQL as {{repro.sql}} and run:
{code:bash}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.shuffle.partitions=2
-f repro.sql
{code}
h2. Expected result
For each group, the two-entry map occurs three times, while the one-entry map
occurs only once. Therefore, {{mode(m)}} should always return the two-entry map.
{code:text}
g | m
--+----------------------
0 | {0 -> a, 100 -> b}
1 | {1 -> a, 101 -> b}
2 | {2 -> a, 102 -> b}
3 | {3 -> a, 103 -> b}
4 | {4 -> a, 104 -> b}
5 | {5 -> a, 105 -> b}
6 | {6 -> a, 106 -> b}
7 | {7 -> a, 107 -> b}
{code}
h2. Actual result
The query can return a one-entry map that occurs only once, instead of the most
frequent map. For example:
{code:text}
g | m
--+----------------------
0 | {0 -> x}
1 | {1 -> a, 101 -> b}
2 | {2 -> a, 102 -> b}
3 | {3 -> a, 103 -> b}
4 | {4 -> a, 104 -> b}
5 | {5 -> a, 105 -> b}
6 | {6 -> a, 106 -> b}
7 | {7 -> a, 107 -> b}
{code}
In this result, {{mode(m)}} selects {{0 -> x}} for group {{0}}, although that
map occurs only once and the other map occurs three times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
