xdcjie created SPARK-19269:
------------------------------
Summary: Scheduler Delay in spark ui
Key: SPARK-19269
URL: https://issues.apache.org/jira/browse/SPARK-19269
Project: Spark
Issue Type: Question
Components: Web UI
Affects Versions: 2.0.2
Reporter: xdcjie
In sparkUI, SchedulerDelay is launchtime(in driver) - starttime (in executor),
but in the code, that it consists of launchtime - starttime and statupdate
(executor update status to driver time).
private[ui] def getSchedulerDelay(
info: TaskInfo, metrics: TaskMetricsUIData, currentTime: Long): Long = {
if (info.finished) {
val totalExecutionTime = info.finishTime - info.launchTime
val executorOverhead = (metrics.executorDeserializeTime +
metrics.resultSerializationTime)
math.max(
0,
totalExecutionTime - metrics.executorRunTime - executorOverhead -
getGettingResultTime(info, currentTime))
} else {
// The task is still running and the metrics like executorRunTime are not
available.
0L
}
}
Could we split "scheduler delay" into "scheduler delay time
(launchtime-starttime) and "status update time". in my workspace, scheduler
delay time is 60s, but launchtime - starttime is very fast, <1s.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
