Hi Akhil, thanks for your help but I was originally running without xmx option. With that I was just trying to push the limit of my heap size, but obviously doing it wrong.
Arian Pasquali http://about.me/arianpasquali 2014-10-20 12:24 GMT+01:00 Akhil Das <ak...@sigmoidanalytics.com>: > Hi Arian, > > You will get this exception because you are trying to create an array that > is larger than the maximum contiguous block of memory in your Java VMs heap. > > Here since you are setting Worker memory as *5Gb* and you are exporting > the *_OPTS as *8Gb*, your application actually thinks it has 8Gb of > memory where as it only has 5Gb and hence it exceeds the VM Limit. > > > > Thanks > Best Regards > > On Mon, Oct 20, 2014 at 4:42 PM, Arian Pasquali <ar...@arianpasquali.com> > wrote: > >> Hi, >> I’m using Spark 1.1.0 and I’m having some issues to setup memory options. >> I get “Requested array size exceeds VM limit” and I’m probably missing >> something regarding memory configuration >> <https://spark.apache.org/docs/1.1.0/configuration.html>. >> >> My server has 30G of memory and this are my current settings. >> >> ##this one seams that was deprecated >> export SPARK_MEM=‘25g’ >> >> ## worker memory options seams to be the memory for each worker (by >> default we have a worker for each core) >> export SPARK_WORKER_MEMORY=‘5g’ >> >> I probably need to specify some options using SPARK_DAEMON_JAVA_OPTS, but >> I’m not quite sure how. >> I have tried some different options like the following, but I still >> couldn’t make it right: >> >> export SPARK_DAEMON_JAVA_OPTS='-Xmx8G -XX:+UseCompressedOops' >> export JAVA_OPTS='-Xmx8G -XX:+UseCompressedOops' >> >> Does anyone has any idea how can I approach this? >> >> >> >> >> 14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator: >> maxBytesInFlight: 50331648, targetRequestSize: 10066329 >> 14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator: >> Getting 1566 non-empty blocks out of 1566 blocks >> 14/10/11 13:00:16 INFO BlockFetcherIterator$BasicBlockFetcherIterator: >> Started 0 remote fetches in 4 ms >> 14/10/11 13:02:06 INFO ExternalAppendOnlyMap: Thread 63 spilling >> in-memory map of 3925 MB to disk (1 time so far) >> 14/10/11 13:05:17 INFO ExternalAppendOnlyMap: Thread 63 spilling >> in-memory map of 3925 MB to disk (2 times so far) >> 14/10/11 13:09:15 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID >> 1566) >> java.lang.OutOfMemoryError: Requested array size exceeds VM limit >> at java.util.Arrays.copyOf(Arrays.java:2271) >> at >> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) >> at >> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) >> at >> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) >> at >> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876) >> at >> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785) >> at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188) >> at >> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) >> at >> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42) >> at >> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73) >> at >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> 14/10/11 13:09:15 ERROR ExecutorUncaughtExceptionHandler: Uncaught >> exception in thread Thread[Executor task launch worker-2,5,main] >> java.lang.OutOfMemoryError: Requested array size exceeds VM limit >> at java.util.Arrays.copyOf(Arrays.java:2271) >> at >> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) >> at >> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) >> at >> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140 >> >> >> Arian >> > >
