Hi Egor,
we recommand you to set SPARK_HOME because many spark configurations can be
handled better in this way.
SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh is valid only when you set
SPARK_HOME.
Since you get JVM error, if you have any JVM related configuration in your
conf/zeppelin-env.sh can you share it?
If you want to change driver memory without setting SPARK_HOME, you can try
ZEPPELIN_INTP_MEM in conf/zeppelin-env.sh file as below:
export ZEPPELIN_INTP_MEM="-Xmx10g" in conf/zeppelin-env.sh file.
FYI, Zeppelin's default ZEPPELIN_INTP_MEM is "-Xmx1024m
-XX:MaxPermSize=512m".
Though Spark's UI won't display "Storage Memory" as 10g, because it will
shows fraction of java heap to use for memory cache.
I tested both with/without SPARK_HOME setting in yarn-client mode, and
Spark UI shows different value.
- With SPARK_HOME setting if I export SPARK_SUBMIT_OPTIONS="--driver-momory
10g", driver storage memory is 5.2G.
- Without SPARK_HOME setting if I export ZEPPELIN_INT_MEM="-Xmx10g",
driver storage memory is 4.8G.
The difference here is that first one starts spark interpreter process by
running spark_submit and second one runs with java.
Hope this helps!
On Thu, Nov 12, 2015 at 8:41 AM, Egor Pahomov <pahomov.e...@gmail.com>
wrote:
> I'm trying to configure driver memory size. So far I tied next parameters:
>
> -
>
> export JAVA_INTP_OPTS="-Xmx10g"
>
> -
>
> export SPARK_SUBMIT_OPTIONS="--driver-memory 10g --executor-memory 10g"
>
> -
>
> " -Xmx30000m \
>
> -
>
> Change SparkInterpreter.java: conf.set("spark.executor.memory", "10g");
>
> conf.set("spark.executor.cores", "2");
> conf.set("spark.driver.memory", "10g");
> conf.set("spark.shuffle.io.numConnectionsPerPeer", "5");
> conf.set("spark.sql.autoBroadcastJoinThreshold", "200483647");
> conf.set("spark.network.timeout", "400s");
> conf.set("spark.driver.maxResultSize", "3g");
> conf.set("spark.sql.hive.convertMetastoreParquet", "false");
> conf.set("spark.kryoserializer.buffer.max", "200m");
> conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
> conf.set("spark.dynamicAllocation.enabled", "true");
> conf.set("spark.shuffle.service.enabled", "true");
> conf.set("spark.dynamicAllocation.minExecutors", "1");
> conf.set("spark.dynamicAllocation.maxExecutors", "30");
> conf.set("spark.dynamicAllocation.executorIdleTimeout", "60s");
> //.set("spark.sql.hive.metastore.version", "1.1.0")
> conf.set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "100s");
>
> -
>
> I tried setting SPARK_HOME, but it didn't even started, failed with
> "Incompatible minimum and maximum heap sizes specified"
>
>
> No matter, what I do I get in logs: "INFO [2015-11-11 14:55:24,453]
> ({sparkDriver-akka.actor.default-dispatcher-14} Logging.scala[logInfo]:59)
> - Registering block manager 192.168.12.121:45057 with 530.0 MB RAM,
> BlockManagerId(driver, 192.168.12.121, 45057)" and on my spark UI:
>
> ​Has anyone faced this problem or knows what to do?
>
> --
>
>
> *Sincerely yoursEgor Pakhomov*
>
