[
https://issues.apache.org/jira/browse/SPARK-61175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-61175:
-----------------------------
Affects Version/s: 3.5.9
4.0.5
4.1.4
4.2.1
4.3.0
> Make rint return a canonical NaN regardless of the JIT tier
> -----------------------------------------------------------
>
> Key: SPARK-61175
> URL: https://issues.apache.org/jira/browse/SPARK-61175
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.3.0, 5.0.0, 4.2.1, 4.0.5, 4.1.4
> Reporter: Max Gekk
> Priority: Major
>
> rint() can return NaNs with different bit patterns for the same input,
> depending on which JIT tier executes java.lang.Math.rint.
> Spark's Rint calls Math.rint in both eval and generated code. On HotSpot, the
> interpreter and C1 run the Java implementation (StrictMath.rint), which
> clears the sign of a NaN, while C2 replaces the call with an intrinsic
> (roundsd on x86) that returns the input NaN with its sign. Measured on JDK
> 25.0.4.1, x86_64 (AMD Zen 5):
> {noformat}
> input bits -Xint / C1 (TieredStopAtLevel=1) C2
> (-XX:-TieredCompilation)
> fff8000000000000 7ff8000000000000 fff8000000000000
> fff0000000000001 7ff8000000000001 fff8000000000001
> fff4000000000000 7ffc000000000000 fffc000000000000
> {noformat}
> -XX:+PrintIntrinsics shows Math::rint as an intrinsic under C2. Every tier
> keeps the payload and quiets a signalling NaN; only the sign differs.
> The bits reach user code: UnsafeRowWriter.write(double) and
> UnsafeRow.setDouble store the double as is, so collect(), typed Datasets,
> Scala/Java/Python UDF arguments and Arrow-based paths (toPandas with Arrow,
> mapInArrow) can observe the sign, e.g. with Double.doubleToRawLongBits,
> Math.copySign or numpy.signbit. Comparisons, grouping, joins, sorting,
> hashing and string output treat all NaNs alike, and the Parquet and ORC
> writers canonicalize NaN (Double.doubleToLongBits), so SQL results and those
> files are not affected.
> Proposed fix: make Rint return the canonical Double.NaN when its result is
> NaN, in both the interpreted and the generated path, leaving every non-NaN
> result unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
