Hi Sim, I think the right way to set the PermGen Size is through driver extra JVM options, i.e.
--conf "spark.driver.extraJavaOptions=-XX:MaxPermSize=256m" Can you try it? Without this conf, your driver's PermGen size is still 128m. Thanks, Yin On Mon, Jul 6, 2015 at 4:07 AM, Denny Lee <denny.g....@gmail.com> wrote: > I went ahead and tested your file and the results from the tests can be > seen in the gist: https://gist.github.com/dennyglee/c933b5ae01c57bd01d94. > > Basically, when running {Java 7, MaxPermSize = 256} or {Java 8, default} > the query ran without any issues. I was able to recreate the issue with > {Java 7, default}. I included the commands I used to start the spark-shell > but basically I just used all defaults (no alteration to driver or executor > memory) with the only additional call was with driver-class-path to connect > to MySQL Hive metastore. This is on OSX Macbook Pro. > > One thing I did notice is that your version of Java 7 is version 51 while > my version of Java 7 version 79. Could you see if updating to Java 7 > version 79 perhaps allows you to use the MaxPermSize call? > > > > > On Mon, Jul 6, 2015 at 1:36 PM Simeon Simeonov <s...@swoop.com> wrote: > >> The file is at >> https://www.dropbox.com/s/a00sd4x65448dl2/apache-spark-failure-data-part-00000.gz?dl=1 >> >> The command was included in the gist >> >> SPARK_REPL_OPTS="-XX:MaxPermSize=256m" >> spark-1.4.0-bin-hadoop2.6/bin/spark-shell --packages >> com.databricks:spark-csv_2.10:1.0.3 --driver-memory 4g --executor-memory 4g >> >> /Sim >> >> Simeon Simeonov, Founder & CTO, Swoop <http://swoop.com/> >> @simeons <http://twitter.com/simeons> | blog.simeonov.com | 617.299.6746 >> >> >> From: Yin Huai <yh...@databricks.com> >> Date: Monday, July 6, 2015 at 12:59 AM >> To: Simeon Simeonov <s...@swoop.com> >> Cc: Denny Lee <denny.g....@gmail.com>, Andy Huang < >> andy.hu...@servian.com.au>, user <user@spark.apache.org> >> >> Subject: Re: 1.4.0 regression: out-of-memory errors on small data >> >> I have never seen issue like this. Setting PermGen size to 256m should >> solve the problem. Can you send me your test file and the command used to >> launch the spark shell or your application? >> >> Thanks, >> >> Yin >> >> On Sun, Jul 5, 2015 at 9:17 PM, Simeon Simeonov <s...@swoop.com> wrote: >> >>> Yin, >>> >>> With 512Mb PermGen, the process still hung and had to be kill -9ed. >>> >>> At 1Gb the spark shell & associated processes stopped hanging and >>> started exiting with >>> >>> scala> println(dfCount.first.getLong(0)) >>> 15/07/06 00:10:07 INFO storage.MemoryStore: ensureFreeSpace(235040) >>> called with curMem=0, maxMem=2223023063 >>> 15/07/06 00:10:07 INFO storage.MemoryStore: Block broadcast_2 stored as >>> values in memory (estimated size 229.5 KB, free 2.1 GB) >>> 15/07/06 00:10:08 INFO storage.MemoryStore: ensureFreeSpace(20184) >>> called with curMem=235040, maxMem=2223023063 >>> 15/07/06 00:10:08 INFO storage.MemoryStore: Block broadcast_2_piece0 >>> stored as bytes in memory (estimated size 19.7 KB, free 2.1 GB) >>> 15/07/06 00:10:08 INFO storage.BlockManagerInfo: Added >>> broadcast_2_piece0 in memory on localhost:65464 (size: 19.7 KB, free: 2.1 >>> GB) >>> 15/07/06 00:10:08 INFO spark.SparkContext: Created broadcast 2 from >>> first at <console>:30 >>> java.lang.OutOfMemoryError: PermGen space >>> Stopping spark context. >>> Exception in thread "main" >>> Exception: java.lang.OutOfMemoryError thrown from the >>> UncaughtExceptionHandler in thread "main" >>> 15/07/06 00:10:14 INFO storage.BlockManagerInfo: Removed >>> broadcast_2_piece0 on localhost:65464 in memory (size: 19.7 KB, free: 2.1 >>> GB) >>> >>> That did not change up until 4Gb of PermGen space and 8Gb for driver & >>> executor each. >>> >>> I stopped at this point because the exercise started looking silly. It >>> is clear that 1.4.0 is using memory in a substantially different manner. >>> >>> I'd be happy to share the test file so you can reproduce this in your >>> own environment. >>> >>> /Sim >>> >>> Simeon Simeonov, Founder & CTO, Swoop <http://swoop.com/> >>> @simeons <http://twitter.com/simeons> | blog.simeonov.com | 617.299.6746 >>> >>> >>> From: Yin Huai <yh...@databricks.com> >>> Date: Sunday, July 5, 2015 at 11:04 PM >>> To: Denny Lee <denny.g....@gmail.com> >>> Cc: Andy Huang <andy.hu...@servian.com.au>, Simeon Simeonov < >>> s...@swoop.com>, user <user@spark.apache.org> >>> Subject: Re: 1.4.0 regression: out-of-memory errors on small data >>> >>> Sim, >>> >>> Can you increase the PermGen size? Please let me know what is your >>> setting when the problem disappears. >>> >>> Thanks, >>> >>> Yin >>> >>> On Sun, Jul 5, 2015 at 5:59 PM, Denny Lee <denny.g....@gmail.com> wrote: >>> >>>> I had run into the same problem where everything was working >>>> swimmingly with Spark 1.3.1. When I switched to Spark 1.4, either by >>>> upgrading to Java8 (from Java7) or by knocking up the PermGenSize had >>>> solved my issue. HTH! >>>> >>>> >>>> >>>> On Mon, Jul 6, 2015 at 8:31 AM Andy Huang <andy.hu...@servian.com.au> >>>> wrote: >>>> >>>>> We have hit the same issue in spark shell when registering a temp >>>>> table. We observed it happening with those who had JDK 6. The problem went >>>>> away after installing jdk 8. This was only for the tutorial materials >>>>> which >>>>> was about loading a parquet file. >>>>> >>>>> Regards >>>>> Andy >>>>> >>>>> On Sat, Jul 4, 2015 at 2:54 AM, sim <s...@swoop.com> wrote: >>>>> >>>>>> @bipin, in my case the error happens immediately in a fresh shell in >>>>>> 1.4.0. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/1-4-0-regression-out-of-memory-errors-on-small-data-tp23595p23614.html >>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>> Nabble.com. >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Andy Huang | Managing Consultant | Servian Pty Ltd | t: 02 9376 >>>>> 0700 | f: 02 9376 0730| m: 0433221979 >>>>> >>>> >>> >>
