HI, I am facing a weird issue. I am using spark 0.9 and running a streaming application.
In the UI, the duration shows order of seconds but if I dig into that particular stage details, it shows total time taken across all tasks for the stage is much much less (in milliseconds) I am using Fair scheduling policy and pool name is counter-metric-persistor. What could the reason for this? *Stage screenshot: Stage 97* 97 counter-metric-persistor<http://204.77.213.186:4040/stages/pool?poolname=counter-metric-persistor> foreach at RealTimeAnalyticsApplication.scala:33<http://204.77.213.186:4040/stages/stage?id=97>2014/05/27 07:22:2314.5 s 6/6 *Stage details screenshot: Stage 97* Details for Stage 97 - *Total task time across all tasks: *154 ms Summary Metrics for 6 Completed Tasks MetricMin 25th percentileMedian 75th percentile Max Result serialization time 0 ms 0 ms 0 ms 0 ms 0 ms Duration 12 ms 13 ms 23 ms 30 ms 54 ms Time spent fetching task results 0 ms 0 ms 0 ms 0 ms 0 ms Scheduler delay 7 ms 7 ms 8 ms 8 ms 8 ms Aggregated Metrics by Executor Executor ID Address Task TimeTotal TasksFailed TasksSucceeded Tasks Shuffle ReadShuffle Write Shuffle Spill (Memory) Shuffle Spill (Disk)0ls230-127-p.nyc0.ls.local:53463199 ms6060.0 B0.0 B0.0 B0.0 B Tasks Task IndexTask ID StatusLocality Level ExecutorLaunch Time DurationGC TimeResult Ser TimeErrors0408SUCCESSPROCESS_LOCALls230-127-p.nyc0.ls.local2014/05/27 07:22:3730 ms 1 411 SUCCESS PROCESS_LOCAL ls230-127-p.nyc0.ls.local 2014/05/27 07:22:3722 ms 2 412 SUCCESS PROCESS_LOCAL ls230-127-p.nyc0.ls.local 2014/05/27 07:22:3723 ms 3 414 SUCCESS PROCESS_LOCAL ls230-127-p.nyc0.ls.local 2014/05/27 07:22:3713 ms 4 415 SUCCESS PROCESS_LOCAL ls230-127-p.nyc0.ls.local 2014/05/27 07:22:3712 ms 5 416 SUCCESS PROCESS_LOCAL ls230-127-p.nyc0.ls.local 2014/05/27 07:22:3754 ms Thanks, -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
