lamber-ken edited a comment on issue #1466: [HUDI-742] Fix Java Math Exception URL: https://github.com/apache/incubator-hudi/pull/1466#issuecomment-606271560 hi, @vinothchandar let me summarize the analysis process #### 1. The exception stackstrace ``` Caused by: java.lang.NoSuchMethodError: java.lang.Math.floorMod(JI)I at org.apache.hudi.index.bloom.BucketizedBloomCheckPartitioner.getPartition(BucketizedBloomCheckPartitioner.java:148) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) ``` #### 2. Check the codebase (BucketizedBloomCheckPartitioner#getPartition) ``` @Override public int getPartition(Object key) { final Pair<String, String> parts = (Pair<String, String>) key; final long hashOfKey = Hashing.md5().hashString(parts.getRight(), StandardCharsets.UTF_8).asLong(); final List<Integer> candidatePartitions = fileGroupToPartitions.get(parts.getLeft()); final int idx = (int) Math.floorMod(hashOfKey, candidatePartitions.size()); assert idx >= 0; return candidatePartitions.get(idx); } ``` #### 3. Analyze 1, The code snippet above seems very normal, we cann't find any problem just only from it. 2, What caused `java.lang.NoSuchMethodError: java.lang.Math.floorMod(JI)I`? #### 4. Some thing about JVM specification from the JVM specification we know that: `I: Integer`, `J:Long`  https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.3.2-200 #### 5. Check released version `0.5.0` / `0.5.1` hudi version 1) parse bytecode ``` jar xf hudi-client-0.5.2-incubating.jar javap -v org/apache/hudi/index/bloom/BucketizedBloomCheckPartitioner.class | grep Math.floor ``` 2) output ``` #14 = Methodref #184.#187 // java/lang/Math.floor:(D)D #64 = Methodref #184.#242 // java/lang/Math.floorMod:(JJ)J 88: invokestatic #14 // Method java/lang/Math.floor:(D)D 54: invokestatic #64 // Method java/lang/Math.floorMod:(JJ)J ``` 3) from the output above, one thing we knew for certain was that official hudi-jar is ok. #### 6. Where is `java.lang.Math.floorMod(JI)I` from? At last we found the answer, @EdwinGuo use `jdk11` build the hudi project and then run on `jdk8`, it may caused by `jdk11` jvm compile mechanism or `floorMod(JI)I` exists in jdk11. #### 7. End So, the changes LGTM, thanks @EdwinGuo
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
