Hi Nick
Maybe if you use:
export SPARK_MEM=4g
On Mon, Jul 21, 2014 at 11:35 AM, Nick R. Katsipoulakis <kat...@cs.pitt.edu>
wrote:
> Hello,
>
> Currently I work on a project in which:
>
> I spawn a standalone Apache Spark MLlib job in Standalone mode, from a
> running Java Process.
>
> In the code of the Spark Job I have the following code:
>
> SparkConf sparkConf = new SparkConf().setAppName("SparkParallelLoad");
> sparkConf.set("spark.executor.memory", "8g");
> JavaSparkContext sc = new JavaSparkContext(sparkConf);
>
> ...
>
> Also, in my ~/spark/conf/spark-env.sh I have the following values:
>
> SPARK_WORKER_CORES=1
> export SPARK_WORKER_CORES=1
> SPARK_WORKER_MEMORY=2g
> export SPARK_WORKER_MEMORY=2g
> SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.spark.executor.memory=4g"
> export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.spark.executor.memory=4g"
>
> During runtime I receive a Java OutOfMemory exception and a Core dump. My
> dataset is less than 1 GB and I want to make sure that I cache it all in
> memory for my ML task.
>
> Am I increasing the JVM Heap Memory correctly? Am I doing something wrong?
>
> Thank you,
>
> Nick
>
>
