[
https://issues.apache.org/jira/browse/SPARK-58928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58928:
-----------------------------------
Labels: pull-request-available (was: )
> WindowGroupLimitExec uses binary equality for collated PARTITION BY keys
> ------------------------------------------------------------------------
>
> Key: SPARK-58928
> URL: https://issues.apache.org/jira/browse/SPARK-58928
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Vinod KC
> Priority: Major
> Labels: pull-request-available
>
> For a query like _RANK() OVER (PARTITION BY s ...)_ with {_}WHERE rank <=
> N{_}, the optimizer adds a WindowGroupLimitExec to drop rows early. It finds
> partition boundaries by comparing keys byte-for-byte (nextGroup ==
> currentGroup).
> Byte comparison is wrong for collated strings: under UTF8_LCASE, 'foo' and
> 'FOO' are equal but have different bytes, so one partition gets split into
> several.
> The final result is still correct (the later WindowExec recomputes the
> ranking), but the limit no longer prunes collated partitions, so it forwards
> many more rows than needed and the optimization stops helping.
> Repro:
> {code:java}
> CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
> INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
> SELECT s, i, r FROM (
> SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
> ) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
>
> {code}
> Need to compare keys the same way WindowExec already does, use byte
> comparison only when all keys are binary-stable, and fall back to
> collation-aware comparison when any key is not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
