Sahil Takiar created HIVE-16484:
-----------------------------------
Summary: Investigate SparkLauncher for HoS as alternative to
bin/spark-submit
Key: HIVE-16484
URL: https://issues.apache.org/jira/browse/HIVE-16484
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar
The {{SparkClientImpl#startDriver}} currently looks for the {{SPARK_HOME}}
directory and invokes the {{bin/spark-submit}} script, which spawns a separate
process to run the Spark application.
{{SparkLauncher}} was added in SPARK-4924 and is a programatic way to launch
Spark applications.
I see a few advantages:
* No need to spawn a separate process to launch a HoS --> lower startup time
* Simplifies the code in {{SparkClientImpl}} --> easier to debug
* {{SparkLauncher#startApplication}} returns a {{SparkAppHandle}} which
contains some useful utilities for querying the state of the Spark job
** It also allows the launcher to specify a list of job listeners
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
