[
https://issues.apache.org/jira/browse/SPARK-59514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59514:
-----------------------------------
Labels: pull-request-available (was: )
> assertDataFrameEqual accepts unequal NaN and infinity values
> ------------------------------------------------------------
>
> Key: SPARK-59514
> URL: https://issues.apache.org/jira/browse/SPARK-59514
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Akshay Thorat
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> The floating-point tolerance inequality does not reject NaN arithmetic. An
> infinite expected value can also make both sides infinity. As a result,
> unequal values pass equality assertions.
> h2. Reproduction
> Reproduced on upstream master 39776477a3d (PySpark 5.0.0.dev0), Python
> 3.10.11. No SparkSession is required.
> {code:python}
> from pyspark.sql import Row
> from pyspark.testing import assertDataFrameEqual
> assertDataFrameEqual([Row(x=float("nan"))], [Row(x=1.0)])
> assertDataFrameEqual([Row(x=1.0)], [Row(x=float("inf"))])
> assertDataFrameEqual([Row(x=float("inf"))], [Row(x=float("-inf"))])
> # All three return without raising
> {code}
> h2. Expected behavior
> Each assertion should raise PySparkAssertionError with condition
> DIFFERENT_ROWS. Matching NaNs and same-sign infinities should continue to
> compare equal.
> h2. Proposed fix and verification
> Compare NaN and infinity explicitly before applying finite-value tolerances.
> Tests cover both argument directions, zero/default tolerances, row-order
> settings, and nested values. Regression tests fail before the fix and pass
> afterward. Python compilation, custom-error checks and Ruff checks pass.
> Pull request: https://github.com/apache/spark/pull/58790
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
