[
https://issues.apache.org/jira/browse/SPARK-59011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109753#comment-18109753
]
Viet Bui commented on SPARK-59011:
----------------------------------
h2. {color:#202124}Hi, our university project team has reproduced this issue
and submitted a fix with a regression test.
*Pull request:* https://github.com/apache/spark/pull/58337
*Contributors:* Phan Trung Nhut and Bui Nam Viet.{color}
> Series.rank with method='first' and ascending=False gives wrong ranks for
> tied values
> -------------------------------------------------------------------------------------
>
> Key: SPARK-59011
> URL: https://issues.apache.org/jira/browse/SPARK-59011
> Project: Spark
> Issue Type: Task
> Components: Pandas API on Spark
> Affects Versions: 4.2.1
> Reporter: Jing Zhan
> Priority: Major
> Labels: pull-request-available
>
> Series.rank(method='first', ascending=False) produces incorrect ranks for
> tied values in the pandas API on Spark. The tiebreaker should always favor
> the element that appeared first in the original array (lower index), but
> Spark incorrectly reverses the tiebreaker direction when ascending=False.
> Reproduction:
> import pandas as pd
> import pyspark.pandas as ps
> pser = pd.Series([1, 2, 3, 1], name='x')
> psser = ps.from_pandas(pser)
> print(pser.rank(method='first', ascending=False).tolist()) # [3.0, 2.0,
> 1.0, 4.0]
> print(psser.rank(method='first', ascending=False).sort_index().tolist()) #
> [4.0, 2.0, 1.0, 3.0]
> Expected: [3.0, 2.0, 1.0, 4.0] (matches pandas — index 0 gets rank 3, index 3
> gets rank 4)
> Actual: [4.0, 2.0, 1.0, 3.0] (wrong — index 0 gets rank 4, index 3 gets rank
> 3)
> Root cause: In Series._rank, the natural order tiebreaker column uses the
> same direction as the value sort (DESC when ascending=False). This reverses
> the tiebreaker for tied values. The fix is to always use ASC for the natural
> order tiebreaker since "first" always means first in the original array,
> regardless of rank direction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
