My Spark streaming application processes the data received in each interval.
In Spark Stages UI, all the stages are pointed to single line of code*
windowDStream.foreachRDD* only (not the actions inside the DStream)
- Following is the information from Spark Stages UI page:
Stage IdDescr
What he meant is that look it up in the Spark UI, specifically in the Stage
tab to see what is taking so long. And yes code snippet helps us debug.
TD
On Fri, Apr 3, 2015 at 12:47 AM, Akhil Das
wrote:
> You need open the Stage\'s page which is taking time, and see how long its
> spending on GC
You need open the Stage\'s page which is taking time, and see how long its
spending on GC etc. Also it will be good to post that Stage and its
previous transformation's code snippet to make us understand it better.
Thanks
Best Regards
On Fri, Apr 3, 2015 at 1:05 PM, Vijay Innamuri
wrote:
>
> Wh
When I run the Spark application (streaming) in local mode I could see the
execution progress as below..
[Stage
0:>
(1817 + 1) / 3125]
[Stage
2:===>
(740 + 1) / 3125]
One of the stages
