Thanks Andrew, I tried and it works.
On Fri, Jul 18, 2014 at 12:53 AM, Andrew Or wrote:
> You will need to include that in the SPARK_JAVA_OPTS environment variable,
> so add the following line to spark-env.sh:
>
> export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC"
>
> This should propagate to th
You will need to include that in the SPARK_JAVA_OPTS environment variable,
so add the following line to spark-env.sh:
export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC"
This should propagate to the executors. (Though you should double check,
since 0.9 is a little old and I could be forgetting some
Thanks Andrew.
Say that I want to turn on CMS gc for each worker.
All I need to do is add the following line to conf/spark-env.sh on node
where I submit the application.
-XX:+UseConcMarkSweepGC
Is that correct?
Will this option be populated to each worker in yarn?
On Thu, Jul 17, 2014 at 9:
Hi Chen,
spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark
0.9. You need to
export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2"
in conf/spark-env.sh.
Let me know if that works.
Andrew
2014-07-17 18:15 GMT-07:00 Tathagata Das :
> Can you check in the
Can you check in the environment tab of Spark web ui to see whether this
configuration parameter is in effect?
TD
On Thu, Jul 17, 2014 at 2:05 PM, Chen Song wrote:
> I am using spark 0.9.0 and I am able to submit job to YARN,
> https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
>
> I am
