Yeah, this is definitely confusing. The motivation for this was that different
users of the same cluster may want to set different memory sizes for their
apps, so we decided to put this setting in the driver. However, if you put
SPARK_JAVA_OPTS in spark-env.sh, it also applies to executors, whic
Jim, I'm starting to document the heap size settings all in one place,
which has been a confusion for a lot of my peers. Maybe you can take a
look at this ticket?
https://spark-project.atlassian.net/browse/SPARK-1264
On Wed, Mar 19, 2014 at 12:53 AM, Jim Blomo wrote:
> To document this, it wo
To document this, it would be nice to clarify what environment
variables should be used to set which Java system properties, and what
type of process they affect. I'd be happy to start a page if you can
point me to the right place:
SPARK_JAVA_OPTS:
-Dspark.executor.memory can by set on the mach
Thanks for the suggestion, Matei. I've tracked this down to a setting
I had to make on the Driver. It looks like spark-env.sh has no impact
on the Executor, which confused me for a long while with settings like
SPARK_EXECUTOR_MEMORY. The only setting that mattered was setting the
system property
Try checking spark-env.sh on the workers as well. Maybe code there is somehow
overriding the spark.executor.memory setting.
Matei
On Mar 18, 2014, at 6:17 PM, Jim Blomo wrote:
> Hello, I'm using the Github snapshot of PySpark and having trouble setting
> the worker memory correctly. I've set
Hello, I'm using the Github snapshot of PySpark and having trouble setting
the worker memory correctly. I've set spark.executor.memory to 5g, but
somewhere along the way Xmx is getting capped to 512M. This was not
occurring with the same setup and 0.9.0. How many places do I need to
configure the m
