I pinged the IBM team to submit a patch that would work on IBM JVM.
On Fri, Aug 14, 2015 at 11:27 AM, Pete Robbins <robbin...@gmail.com> wrote: > ref: https://issues.apache.org/jira/browse/SPARK-9370 > > The code to handle BigInteger types in > > org.apache.spark.sql.catalyst.expressions.UnsafeRowWriters.java > > and > > org.apache.spark.unsafe.Platform.java > > is dependant on the implementation of java.math.BigInteger > > eg: > > try { > signumOffset = > _UNSAFE.objectFieldOffset(BigInteger.class.getDeclaredField("signum")); > magOffset = > _UNSAFE.objectFieldOffset(BigInteger.class.getDeclaredField("mag")); > } catch (Exception ex) { > // should not happen > } > > This is relying on there being fields "int signum" and "int[] mag" > > These implementaton fields are not part of the Java specification for this > class so can not be relied upon. > > We are running Spark on IBM jdks and their spec-compliant implementation > has different internal fields. This causes an abort when running on these > java runtimes. There is also no guarantee that any future implentations of > OpenJDK will maintain these field names. > > I think we need to find an implementation of these Spark functions that > only relies on Java compliant classes rather than specific implementations. > > Any thoughts? > > Cheers, > > > > > >
