[
https://issues.apache.org/jira/browse/SPARK-58096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095672#comment-18095672
]
Eric Yang commented on SPARK-58096:
-----------------------------------
I'm working on a PR to fix it
> approx_top_k returns wrong counts for ICU-collated string columns (distinct
> non-ASCII values merged)
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-58096
> URL: https://issues.apache.org/jira/browse/SPARK-58096
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Eric Yang
> Priority: Major
>
> Over an ICU collation (UNICODE, UNICODE_CI, or a locale collation),
> approx_top_k counts DISTINCT non-ASCII string values as a single item. It
> returns fewer items than actually exist, with inflated (summed) counts – a
> silent wrong result for the frequency estimates.
> *Repro* (three distinct characters, so the correct top-k has 3 items with
> counts 3, 2, 1):
> {code:sql}
> SELECT approx_top_k(c, 10)
> FROM (SELECT CAST(col AS STRING COLLATE UNICODE_CI) AS c
> FROM VALUES ('且'),('且'),('且'),('丕'),('丕'),('世') AS t(col));
> -- returns a SINGLE item with count 6, instead of 3 items with counts 3, 2, 1
> {code}
> Impact: wrong frequency counts and missing distinct items for any non-ASCII
> data under
> UNICODE / UNICODE_CI / locale collations. UTF8_BINARY and UTF8_LCASE are not
> affected.
> {*}Note{*}: the fix should assure that the key and original must round-trip
> losslessly through update → serialize → deserialize (no Java String
> intermediary)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
