I've never seen the ReplSuite test OoMing with IBM's latest SDK for Java but have always noticed this particular test failing with the following instead:
java.lang.AssertionError: assertion failed: deviation too large: 0.8506807397223823, first size: 180392, second size: 333848 This particular test could be improved and I don't think it should hold up releases, I've commented on [SPARK-14558] already a while back and the discussion ended with: "A better check would be to run with and without the closure cleaner change -> Yea, this is what I did locally, but how to write a test for it?" It will fail in this particular way reliably with Open/Oracle JDK as well if you were to use Kryo. We don't see this test failing (either OoM or the above failure) with OpenJDK 8 in our test farm, this is with OpenJDK 1.8.0_51-b16 and I'm running with -Xmx4g -Xss2048k -Dspark.buffer.pageSize 1048576. All other Spark unit tests pass (we see a grand total of 11980 tests) except for the Kafka stress test already mentioned, various platforms/operating systems including big-endian. I've never seen the NoSuchMethod error mentioned in JavaUDFSuite and haven't seen the failure Alan mentions below either. I also have performance data to share (HiBench and SparkSqlPerf with TPC-DS queries) comparing this release to Spark 2.0.2, I'll wait until the next RC before commenting (it is positive), looks like we'll have another as this RC2 vote should be closed by now and in RC3 we'd also have the [SPARK-18091] fix included to prevent a test's generated code exceeding the 64k constant pool size limit. From: akchin <akc...@us.ibm.com> To: dev@spark.apache.org Date: 13/12/2016 19:51 Subject: Re: [VOTE] Apache Spark 2.1.0 (RC2) Hello, I am seeing this error as well except during "define case class and create Dataset together with paste mode *** FAILED ***" Starts throwing OOM and GC errors after running for several minutes. ----- Alan Chin IBM Spark Technology Center -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Apache-Spark-2-1-0-RC2-tp20175p20215.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
