Thanks Andrew, I tried and it works.
On Fri, Jul 18, 2014 at 12:53 AM, Andrew Or <and...@databricks.com> wrote: > You will need to include that in the SPARK_JAVA_OPTS environment variable, > so add the following line to spark-env.sh: > > export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC" > > This should propagate to the executors. (Though you should double check, > since 0.9 is a little old and I could be forgetting something) If you wish > to add spark options in addition to this, simply append them to the > environment variable: > > export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC -Dspark.config.one=value > -Dspark.config.two=value" > > (Please note that this is only for Spark 0.9. The part where we set Spark > options within SPARK_JAVA_OPTS is deprecated as of 1.0) > > > 2014-07-17 21:08 GMT-07:00 Chen Song <chen.song...@gmail.com>: > > Thanks Andrew. >> >> Say that I want to turn on CMS gc for each worker. >> >> All I need to do is add the following line to conf/spark-env.sh on node >> where I submit the application. >> >> -XX:+UseConcMarkSweepGC >> >> Is that correct? >> >> Will this option be populated to each worker in yarn? >> >> >> >> On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or <and...@databricks.com> wrote: >> >>> Hi Chen, >>> >>> spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark >>> 0.9. You need to >>> >>> export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2" >>> >>> in conf/spark-env.sh. >>> >>> Let me know if that works. >>> Andrew >>> >>> >>> 2014-07-17 18:15 GMT-07:00 Tathagata Das <tathagata.das1...@gmail.com>: >>> >>> Can you check in the environment tab of Spark web ui to see whether this >>>> configuration parameter is in effect? >>>> >>>> TD >>>> >>>> >>>> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <chen.song...@gmail.com> >>>> wrote: >>>> >>>>> I am using spark 0.9.0 and I am able to submit job to YARN, >>>>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html. >>>>> >>>>> I am trying to turn on gc logging on executors but could not find a >>>>> way to set extra Java opts for workers. >>>>> >>>>> I tried to set spark.executor.extraJavaOptions but that did not work. >>>>> >>>>> Any idea on how I should do this? >>>>> >>>>> -- >>>>> Chen Song >>>>> >>>>> >>>> >>> >> >> >> -- >> Chen Song >> >> > -- Chen Song
