Application not found in Spark historyserver in yarn-client mode

[Anfernee Xu]

Sorry, I have to re-send it again as I did not get the answer.

Here's the problem I'm facing, I'm using Spark 1.5.0 release, I have a
standalone java application which is periodically submit Spark jobs to my
yarn cluster, btw I'm not using 'spark-submit' or
'org.apache.spark.launcher' to submit my jobs. These jobs are successful
and I can see them on Yarn RM webUI, but when I want to follow the link to
the app history on Spark historyserver, I always got 404(application is not
found) from Spark historyserver.

My code looks likes as below


    SparkConf conf = new
SparkConf().setAppName("testSpak").setMaster("yarn-client")
        .setJars(new String[]{IOUtil.getJar(MySparkApp.class)});

    conf.set("spark.yarn.historyServer.address", "10.247.44.155:18080");
    conf.set("spark.history.fs.logDirectory",
        "
hdfs://myHdfsNameNode:55310/scratch/tie/spark/applicationHistory");

    JavaSparkContext sc = new JavaSparkContext(conf);

    try {

     ... my application code

    }finally{
      sc.stop();

    }

Anything I did wrong or missed? Do I need to configure something on Yarn
side?

Thanks
-- 
--Anfernee