GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2696
[SPARK-2321] [WIP] Stable pull-based progress / status API
This is a WIP pull request that proposes a stable, pull-based progress /
status API for Spark (see
[SPARK-2321](https://issues.apache.org/jira/browse/SPARK-2321)). For now, I'd
like to discuss the basic implementation, API names, and overall interface
design. Once we arrive at a good design, I'll go back and add additional
methods to expose more information via these API.
#### Design goals:
- Pull-based API
- Usable from Java / Scala / Python
- Can be extended to expose more information without introducing binary
incompatibilities.
- Returns immutable objects.
- Don't leak any implementation details, preserving our freedom to change
the implementation.
#### Implementation:
- Add public methods (`getJobInfo`, `getStageInfo`) to SparkContext to
allow status / progress information to be retrieved.
- Add public interfaces (`SparkJobInfo`, `SparkStageInfo`) for our API
return values. These interfaces consist entirely of Java-style getter methods.
The interfaces are currently implemented in Java. I decided to explicitly
separate the interface from its implementation (`SparkJobInfoImpl`,
`SparkStageInfoImpl`) in order to prevent users from constructing these
responses themselves.
- Move the construction of the web UI's SparkListeners into SparkContext
and out of the web UI tabs themselves. This allows us to re-use these
listeners in the implementation of this status API. There are a few reasons
why this listener re-use makes sense:
- The status API and web UI are guaranteed to show consistent
information.
- These listeners are already well-tested.
- The same garbage-collection / information retention configurations can
apply to both this API and the web UI.
- Extend JobProgressListener to maintain `jobId -> Job` and `stageId ->
Stage` mappings.
#### Open questions:
- I'm open to naming suggestions / improvements.
- There's a "job -> stages -> tasks" hierarchy; how should we expose this
in the API responses? Should `SparkJobInfo` only expose stages' ids, requiring
consumers to call `getStageInfo` to get information on stages? If it returns
`SparkStageInfo` objects, then should it enforce "consistent snapshot"
semantics such that once I get a `SparkJobInfo` I have a complete, immutable
snapshot of the state of the job's stages and tasks at the time that I call
`getJobInfo()`? That would be nice, but potentially expensive for large jobs.
- By implementing the interfaces in Java, we're limited in terms of how
much Scala syntactic sugar we can provide for Scala users. For example,
paren-less accessor methods don't seem to work on Java interfaces, so this
current API means that Scala users have to write
```scala
sc.getJobInfo(0).getStages()(0)
```
to get information on the first stage in a job. On the other hand, this
prevents us from introducing incompatibilities if we ever choose to overload
the `getStages` method (or any other ones).
#### TODOs:
- [ ] Unit tests
- [ ] Examples
- [ ] Python API
- [ ] Documentation of new configuration options
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark progress-reporting-api
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2696.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2696
----
commit 24de26347c4421d54393dc60263e462a2112a10b
Author: Josh Rosen <[email protected]>
Date: 2014-10-07T04:02:14Z
Create UI listeners in SparkContext instead of in Tabs:
This change means that the listeners will always be registered, even if the
web
UI is disabled. The motivation for this change is to allow these listeners
to
be used when implementing a stable pull-based status / progress API for 1.2.
commit ac2d13aae4fc226c9e625ae6a9e6bba0f71d2dd6
Author: Josh Rosen <[email protected]>
Date: 2014-10-07T06:23:56Z
Add jobId->stage, stageId->stage mappings in JobProgressListener
commit 08cbec9e7f5b38e40c7ab7c41782fef67db74024
Author: Josh Rosen <[email protected]>
Date: 2014-10-07T06:24:35Z
Begin to sketch the interfaces for a stable, public status API.
Some design goals here:
- Hide constructors and implementations from users; only expose interfaces.
- Return only immutable objects.
- Ensure API will be usable from Java (e.g. only expose Array collections,
since they're nicely usable from Scala and Java without having to do any
implicit conversions on the Scala side or wrapping into Java-friendly
types on the Java side).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
