Hi All,
I have HistoryServer up and running, and it is great.
Is it possible to also enable HsitoryServer to parse failed jobs event by
default as well?
I get "No Completed Applications Found" if job fails.
=========Event Log Location: hdfs:///user/test01/spark/logs/No Completed
Applications Found=========
The reason is that it is good to run the HistoryServer to keep track of
performance and resource usage for each completed job, but I found it more
useful when job fails. I can identify which stage did it fail, etc instead of
sipping through the logs from the Resource Manager. The same event log is only
available when the Application Master is still active, once the job fails, the
Application Master is killed, and I lose the GUI access, even though I have the
event log in JSON format, I can't open it with the HistoryServer.
This is very helpful especially for long running jobs that last for 2-18 hours
that generates Gigabytes of logs.
So I have 2 questions:
1. Any reason why we only render completed jobs? Why can't we bring in all jobs
and choose from the GUI? Like a time machine to restore the status from the
Application Master?
./core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
val logInfos = logDirs
.sortBy { dir => getModificationTime(dir) }
.map { dir => (dir,
EventLoggingListener.parseLoggingInfo(dir.getPath, fileSystem)) }
.filter { case (dir, info) => info.applicationComplete }
2. If I force to touch a file "APPLICATION_COMPLETE" in the failed job event
log folder, will this cause any problem?
