Nilanjan Roy created ZEPPELIN-5397:
--------------------------------------
Summary: SPARK interpreter not starting
Key: ZEPPELIN-5397
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5397
Project: Zeppelin
Issue Type: Bug
Components: interpreter-launcher, spark
Affects Versions: 0.9.0
Environment: We are using docker to run zeppelin. Both zeppelin and
spark are installed inside the same container running debian buster. We are
running mesos in a separate cluster and setting SPARK_MASTER to point to the
mesos cluster.
Reporter: Nilanjan Roy
Attachments: interpreter.sh
We are setting following SPARK_SUBMIT_OPTIONS in zeppelin-env.sh file.
{code:java}
export SPARK_SUBMIT_OPTIONS="$SPARK_SUBMIT_OPTIONS --conf
'spark.driver.extraJavaOptions=-Dcom.example.env=production
-Dcom.example.role=zeppelin'"
export SPARK_SUBMIT_OPTIONS="$SPARK_SUBMIT_OPTIONS --conf
'spark.executor.extraJavaOptions=-Dcom.example.env=production
-Dcom.example.role=zeppelin -Dfile.encoding=UTF-8'"
{code}
The spark interpreter is failing to start with error
{code:java}
Error: Unrecognized option: -Dcom.example.role='
{code}
We investigated the issue and the issue seems to be in the bin/interpreter.sh
script which is failing to parse the SPARK_SUBMIT_OPTIONS properly. When the
INTERPRETER_RUN_COMMAND is expanded we can see how the SPARK_SUBMIT_OPTIONS is
interpreted
{code:java}
--conf ''\''spark.driver.extraJavaOptions=-Dcom.example.env=production'
'-Dcom.example.role=zeppelin'\''' --conf
''\''spark.executor.extraJavaOptions=-Dcom.example.env=production'
-Dcom.example.role=zeppelin '-Dfile.encoding=UTF-8'\'''
{code}
Upon further investigation we found the issue is the whitespace between the
Dcom.example.env=production and Dcom.example.role in
spark.driver.extraJavaOptions ** and similarly in
spark.executor.extraJavaOptions.
This issue was not there in a SNAPSHOT version of 0.9.0 (not able to find that
anymore). Attaching the interpreter.sh from that version which is
working.[^interpreter.sh]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
