Application not showing in Spark History

[Rychnovsky, Dusan]

Hi,


I am trying to launch my Spark application from within my Java application via 
the SparkSubmit class, like this:


List<String> args = new ArrayList<>();

args.add("--verbose");
args.add("--deploy-mode=cluster");
args.add("--master=yarn");
...


SparkSubmit.main(args.toArray(new String[args.size()]));



This works fine, with one catch - the application does not appear in Spark 
History after it's finished.


If, however, I run the application using `spark-submit.sh`, like this:



spark-submit \

  --verbose \

  --deploy-mode=cluster \

  --master=yarn \

  ...



the application appears in Spark History correctly.


What am I missing?


Also, is this a good way to launch a Spark application from within a Java 
application or is there a better way?

Thanks,

Dusan