Hello Jongyoul
Excellent write up to demystify the execution life-cycle of a paragraph.
There is a small typo in the blog: "At last, it has two words separated by
comma, *getInterpreter* handles it as “%{group_name}.{interpreter_name}”"
Instead of comma, shouldn't it be DOT (.) ?
There is also another unknown details that may be interesting to expose:
how are interpreter JVM launched ?
InterpreterFactory
private Interpreter createRemoteRepl(String interpreterPath, String
noteId, String className,
Properties property, String interpreterSettingId) {
int connectTimeout =
conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" +
interpreterSettingId;
int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
RemoteInterpreter remoteInterpreter =
new RemoteInterpreter(property, noteId, className,
conf.getInterpreterRemoteRunnerPath(),
interpreterPath, localRepoPath, connectTimeout, maxPoolSize,
remoteInterpreterProcessListener, appEventListener);
remoteInterpreter.addEnv(env);
return new LazyOpenInterpreter(remoteInterpreter);
}
The code will call the bin/intepreter.sh (or interpreter.cmd if on Windows)
to launch a new JVM for the interpreter.
And depending on the type of interpreter, the command will be different:
if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path
"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options
"${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp
${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER}
${PORT} &
fi
There is a special case for Spark because we execute a SparkSubmit.
This detail is important for contributors because when they need to DEBUG
an interpreter code, they have to modify the interpreter.sh file to add an
"-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxx"
Most of people naively add this debug argument to the zeppelin.sh file but
it only debugs the Zeppelin server itself, not the interpreter.
Do you think we may create a page in the official doc for this ?
On Tue, Oct 4, 2016 at 9:00 AM, Jongyoul Lee <jongy...@gmail.com> wrote:
> Hello devs and users,
>
> I've submitted a post into Apache Zeppelin Stories on medium.com:
> https://medium.com/apache-zeppelin-stories/how-apache-zeppelin-runs-a-
> paragraph-783a0a612ba9#.x9ym8fovv
>
> I hope all users and contributors understand Apache Zeppelin deeply.
>
> Regards,
> Jongyoul
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
