[
https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhenyu Zheng updated HIVE-23133:
--------------------------------
Description:
Use java.lang.StrictMath rather than java.lang.Math in UDFs for compatibility:
UDFCos.java
UDFExp.java
UDFLn.java
UDFLog.java
UDFLog2.java
UDFLog10.java
Currently, we have set up an ARM CI to test out how Hive works on ARM platform:
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/]
Among the failures, we have observed that some numeric operations can have
different result across hardware archs, such as:
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/]
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/]
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/]
we can see that the calculation results of log, exp, cos, toRadians etc is
slitly different than the .out file results that we are
comparing(they are tested and wrote on X86 machines), this is because of we use
[Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]
for these kind of calculations.
and according to the
[illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]:
_Unlike some of the numeric methods of class StrictMath, all implementations of
the equivalent functions of class Math are not_
_defined to return the bit-for-bit same results. This relaxation permits
better-performing implementations where strict reproducibility_
_is not required._
_By default many of the Math methods simply call the equivalent method in
StrictMath for their implementation._
_Code generators are encouraged to use platform-specific native libraries or
microprocessor instructions, where available,_
_to provide higher-performance implementations of Math methods._
so the result will have difference across hardware archs.
On the other hand, JAVA provided another library
[StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]
that will not have this kind of problem as according to its'
[reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]:
_To help ensure portability of Java programs, the definitions of some of the
numeric functions in this package require that they produce_
_the same results as certain published algorithms._
So in order to fix the above mentioned problem, we have to consider switch to
use StrictMath instead of Math.
was:
Use java.lang.StrictMath rather than java.lang.Math in UDFs:
UDFCos.java
UDFExp.java
UDFLn.java
UDFLog.java
UDFLog2.java
UDFLog10.java
Currently, we have set up an ARM CI to test out how Hive works on ARM platform:
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/]
Among the failures, we have observed that some numeric operations can have
different result across hardware archs, such as:
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/]
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/]
[https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/]
we can see that the calculation results of log, exp, cos, toRadians etc is
slitly different than the .out file results that we are
comparing(they are tested and wrote on X86 machines), this is because of we use
[Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]
for these kind of calculations.
and according to the
[illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]:
_Unlike some of the numeric methods of class StrictMath, all implementations of
the equivalent functions of class Math are not_
_defined to return the bit-for-bit same results. This relaxation permits
better-performing implementations where strict reproducibility_
_is not required._
_By default many of the Math methods simply call the equivalent method in
StrictMath for their implementation._
_Code generators are encouraged to use platform-specific native libraries or
microprocessor instructions, where available,_
_to provide higher-performance implementations of Math methods._
so the result will have difference across hardware archs.
On the other hand, JAVA provided another library
[StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]
that will not have this kind of problem as according to its'
[reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]:
_To help ensure portability of Java programs, the definitions of some of the
numeric functions in this package require that they produce_
_the same results as certain published algorithms._
So in order to fix the above mentioned problem, we have to consider switch to
use StrictMath instead of Math.
> Numeric operations can have different result across hardware archs
> ------------------------------------------------------------------
>
> Key: HIVE-23133
> URL: https://issues.apache.org/jira/browse/HIVE-23133
> Project: Hive
> Issue Type: Sub-task
> Reporter: Zhenyu Zheng
> Assignee: Zhenyu Zheng
> Priority: Major
> Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch,
> HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch
>
>
> Use java.lang.StrictMath rather than java.lang.Math in UDFs for compatibility:
> UDFCos.java
> UDFExp.java
> UDFLn.java
> UDFLog.java
> UDFLog2.java
> UDFLog10.java
>
> Currently, we have set up an ARM CI to test out how Hive works on ARM
> platform:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/]
> Among the failures, we have observed that some numeric operations can have
> different result across hardware archs, such as:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/]
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/]
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/]
> we can see that the calculation results of log, exp, cos, toRadians etc is
> slitly different than the .out file results that we are
> comparing(they are tested and wrote on X86 machines), this is because of we
> use [Math
> Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for
> these kind of calculations.
> and according to the
> [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]:
> _Unlike some of the numeric methods of class StrictMath, all implementations
> of the equivalent functions of class Math are not_
> _defined to return the bit-for-bit same results. This relaxation permits
> better-performing implementations where strict reproducibility_
> _is not required._
> _By default many of the Math methods simply call the equivalent method in
> StrictMath for their implementation._
> _Code generators are encouraged to use platform-specific native libraries or
> microprocessor instructions, where available,_
> _to provide higher-performance implementations of Math methods._
> so the result will have difference across hardware archs.
> On the other hand, JAVA provided another library
> [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]
> that will not have this kind of problem as according to its'
> [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]:
> _To help ensure portability of Java programs, the definitions of some of the
> numeric functions in this package require that they produce_
> _the same results as certain published algorithms._
> So in order to fix the above mentioned problem, we have to consider switch to
> use StrictMath instead of Math.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
