Vinod KC created SPARK-58928:
--------------------------------
Summary: WindowGroupLimitExec uses binary equality for collated
PARTITION BY keys
Key: SPARK-58928
URL: https://issues.apache.org/jira/browse/SPARK-58928
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Vinod KC
For a query like _RANK() OVER (PARTITION BY s ...)_ with {_}WHERE rank <= N{_},
the optimizer adds a WindowGroupLimitExec to drop rows early. It finds
partition boundaries by comparing keys byte-for-byte (nextGroup ==
currentGroup).
Byte comparison is wrong for collated strings: under UTF8_LCASE, 'foo' and
'FOO' are equal but have different bytes, so one partition gets split into
several.
The final result is still correct (the later WindowExec recomputes the
ranking), but the limit no longer prunes collated partitions, so it forwards
many more rows than needed and the optimization stops helping.
Repro:
{code:java}
CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
SELECT s, i, r FROM (
SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
{code}
Need to compare keys the same way WindowExec already does, use byte comparison
only when all keys are binary-stable, and fall back to collation-aware
comparison when any key is not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
