Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18856537
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -853,6 +873,12 @@ class SparkContext(config: SparkConf) extends Logging {
/** The version of Spark on which this application is running. */
def version = SPARK_VERSION
+ def getJobsIdsForGroup(jobGroup: String): Array[Int] =
statusApi.jobIdsForGroup(jobGroup)
+
+ def getJobInfo(jobId: Int): Option[SparkJobInfo] =
statusApi.newJobInfo(jobId)
--- End diff --
A question about the semantics of using all these methods together: is data
about all jobs kept in memory for the whole duration of the job, or is it
discarded after a job or job group is finished?
Asking it in a different way: is it possible for someone to call
`getJobsIdsForGroup` (should that be `getJobIdsForGroup`?) and, when calling
this method to get each job's info, hit an error because the job data has been
discarded?
If that's possible, perhaps we need a different approach, where you get a
snapshot of the whole state of a job / job group in a single method, instead of
having separate finer-grained methods. Or some way to lock things so that the
data is not discarded while someone is looking for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
