RE: How to submit spark job to YARN from scala code

[Forest Fang]

Maybe I'm not understanding your question correctly but would it be possible 
for you to piece up your job submission information as if you are operating 
spark-submit? If so, you could just call  org.apache.spark.deploy.SparkSubmit 
and pass your regular spark-submit arguments.
This is how I do it with my sbt plugin which allows you to codify a 
spark-submit command in sbt build so the JAR gets automatically rebuilt and 
potentially redeployed every time you submit a Spark job using a custom sbt 
task: 
https://github.com/saurfang/sbt-spark-submit/blob/master/src/main/scala/sbtsparksubmit/SparkSubmitPlugin.scala#L85

Subject: Re: How to submit spark job to YARN from scala code
From: ste...@hortonworks.com
CC: user@spark.apache.org
Date: Thu, 17 Dec 2015 19:45:16 +0000










On 17 Dec 2015, at 16:50, Saiph Kappa <saiph.ka...@gmail.com> wrote:


Hi,



Since it is not currently possible to submit a spark job to a spark cluster 
running in standalone mode (cluster mode - it's not currently possible to 
specify this deploy mode within the code), can I do it with YARN?




I tried to do something like this (but in scala): 



«

... // Client object - main method
System.setProperty("SPARK_YARN_MODE", "true")
val sparkConf = new SparkConf()

try {
  val args = new ClientArguments(argStrings, sparkConf)
  new Client(args, sparkConf).run()
} catch {
  case e: Exception => {
    Console.err.println(e.getMessage)
    System.exit(1)
  }
}

System.exit(0)


» in http://blog.sequenceiq.com/blog/2014/08/22/spark-submit-in-java/


However it is not possible to create a new instance of Client since import 
org.apache.spark.deploy.yarn.Client is private








the standard way to work around a problem like this is to place your code in a 
package which has access. File a JIRA asking for a public API too
—one that doesn't require you to set system properties as a way of passing 
parameters down








Is there any way I can submit spark jobs from the code in cluster mode and not 
using the spark-submit script?


Thanks.