Just wondering - how are you launching your application? If you want
to set values like this the right way is to add them to the SparkConf
when you create a SparkContext.
val conf = new SparkConf().set("spark.akka.frameSize",
"10000").setAppName(...).setMaster(...)
val sc = new SparkContext(conf)
- Patrick
On Wed, May 14, 2014 at 9:09 AM, Koert Kuipers <ko...@tresata.com> wrote:
> i have some settings that i think are relevant for my application. they are
> spark.akka settings so i assume they are relevant for both executors and my
> driver program.
>
> i used to do:
> SPARK_JAVA_OPTS="-Dspark.akka.frameSize=10000"
>
> now this is deprecated. the alternatives mentioned are:
> * some spark-submit settings which are not relevant to me since i do not use
> spark-submit (i launch spark jobs from an existing application)
> * spark.executor.extraJavaOptions to set -X options. i am not sure what -X
> options are, but it doesnt sound like what i need, since its only for
> executors
> * SPARK_DAEMON_OPTS to set java options for standalone daemons (i.e. master,
> worker), that sounds like i should not use it since i am trying to change
> settings for an app, not a daemon.
>
> am i missing the correct setting to use?
> should i do -Dspark.akka.frameSize=10000 on my application launch directly,
> and then also set spark.executor.extraJavaOptions? so basically repeat it?
