I'm working on this patch to visualize stages:
https://github.com/apache/spark/pull/2077
Phuoc Do
On Mon, Aug 4, 2014 at 10:12 PM, Zongheng Yang wrote:
> I agree that this is definitely useful.
>
> One related project I know of is Sparkling [1] (also see talk at Spark
> Summit 2014 [2]), but
I agree that this is definitely useful.
One related project I know of is Sparkling [1] (also see talk at Spark
Summit 2014 [2]), but it'd be great (and I imagine somewhat
challenging) to visualize the *physical execution* graph of a Spark
job.
[1] http://pr01.uml.edu/
[2]
http://spark-summit.org
One way to do that is to use RDD.toDebugString to check the dependency
graph and it also gives a good idea regarding stages.
On Mon, Aug 4, 2014 at 8:55 PM, rpandya wrote:
> Is there a way to visualize the task dependency graph of an application,
> during or after its execution? The list of sta
