Re: Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

I got rid of most of my heap errors by increasing the number of partitions of my RDDs by 8-16x. I found in the tuning page that heap space errors can be caused by a hash table that's generated during the shuffle functions, so by splitting up how

Re: Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

Am I trying to reduce it to the minimum number of partitions, or increase the number of partitions with that change? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-java-lang-outOfMemoryError-Java-Heap-Space-tp7735p7739.html Sent from the Apache

Re: Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

Try repartitioning the RDD using coalsce(int partitions) before performing any transforms. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-java-lang-outOfMemoryError-Java-Heap-Space-tp7735p7736.html Sent from the Apache Spark User List mailing li

Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

I've been trying to figure out how to increase the heap space for my spark environment in 1.0.0, and all of the things I've found tell me I have export something in Java Opts, which is deprecated in 1.0.0, or in increase the spark.executor.memory, which is at 6G. I'm only trying to process about 40

Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

I've been trying to figure out how to increase the heap space for my spark environment in 1.0.0, and all of the things I've found tell me I have export something in Java Opts, which is deprecated in 1.0.0, or in increase the spark.executor.memory, which is at 6G. I'm only trying to process about 40