Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2337#issuecomment-56280786
My initial thought was that a "job group"-based approach might be a bit
cleaner, but there are a few subtleties with that proposal that we need to
consider.
What if we had an API that accepts a "job group" and returns the ids of
jobs in that group? e.g.
```scala
// in SparkContext:
def getJobsForGroup(jobGroup: String): Seq[Int]
```
What should we return here? The list of _active_ jobs? All jobs,
including failed and completed ones?
It looks like this pull request addresses a case where you run some
asynchronous action and want to retrieve the ids of all jobs associated with
that action. If `getJobsForGroup` returned all jobs, then it would be the
caller's responsibility to "diff" the results before and after invoking the
action to figure out which jobs were created as a result of that action. This
sort of works if you only have one thread launching jobs in that job group; if
you had multiple threads, then you wouldn't be able to separate the jobs added
by each thread (short of using a unique job group per-thread). There might be
use-cases where I want to fire off a bunch of asynchronous actions (using these
async. APIs) then periodically poll the status of all my FutureActions and it
seems like this could get really messy if I had to switch the job group before
launching each action.
Another subtle problem with this `getJobsForGroup` method: eventually,
we'll have to garbage-collect entries from the jobGroup to jobIds mapping,
which could lead to weird results when diffing two return values.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
