As documented in
http://spark.apache.org/docs/latest/configuration.html#available-properties,
Note for “spark.driver.memory”:
Note: In client mode, this config must not be set through the SparkConf
directly in your application, because the driver JVM has already started at
that point. Instead, please set this through the --driver-memory command line
option or in your default properties file.
If you are to start a SparkR shell using bin/sparkR, then you can use
bin/sparkR –driver-memory. You have no chance to set the driver memory size
after the R shell has been launched via bin/sparkR.
Buf if you are to start a SparkR shell manually without using bin/sparkR (for
example, in Rstudio), you can:
library(SparkR)
Sys.setenv("SPARKR_SUBMIT_ARGS" = "--conf spark.driver.memory=2g sparkr-shell")
sc <- sparkR.init()
From: Dirceu Semighini Filho [mailto:[email protected]]
Sent: Friday, October 23, 2015 7:53 PM
Cc: user
Subject: Re: How to set memory for SparkR with master="local[*]"
Hi Matej,
I'm also using this and I'm having the same behavior here, my driver has only
530mb which is the default value.
Maybe this is a bug.
2015-10-23 9:43 GMT-02:00 Matej Holec
<[email protected]<mailto:[email protected]>>:
Hello!
How to adjust the memory settings properly for SparkR with master="local[*]"
in R?
*When running from R -- SparkR doesn't accept memory settings :(*
I use the following commands:
R> library(SparkR)
R> sc <- sparkR.init(master = "local[*]", sparkEnvir =
list(spark.driver.memory = "5g"))
Despite the variable spark.driver.memory is correctly set (checked in
http://node:4040/environment/), the driver has only the default amount of
memory allocated (Storage Memory 530.3 MB).
*But when running from spark-1.5.1-bin-hadoop2.6/bin/sparkR -- OK*
The following command:
]$ spark-1.5.1-bin-hadoop2.6/bin/sparkR --driver-memory 5g
creates SparkR session with properly adjustest driver memory (Storage Memory
2.6 GB).
Any suggestion?
Thanks
Matej
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-set-memory-for-SparkR-with-master-local-tp25178.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]<mailto:[email protected]>
For additional commands, e-mail:
[email protected]<mailto:[email protected]>
