Hello,
I'm using Spark 2.1.0 and hadoop 2.2.0. When I launch jobs on Yarn, I can retrieve their information on Spark History Server, except that the links to stdout/stderr of executors are wrong -> they lead to their url while the job was running. We have the flag 'yarn.log-aggregation-enable' set to true and once a job is finished on Yarn, its logs are sent to HDFS. On the client end, when I launch my job i set 'spark.eventLog.enabled' to true, and specify 'spark.eventLog.dir'. I can retrieve the DAG and such afterward on Spark History Server. I checked http://spark.apache.org/docs/latest/running-on-yarn.html and http://spark.apache.org/docs/latest/monitoring.html But I do not find what i'm missing to let Spark History Server redirect me to Yarn History Server with a valid link, to see the stdout/stderr logs of the executors. Any idea? Regards, Yohann
