Hi TD, We noticed that Spark Streaming UI is reporting a different task duration from time to time.
e.g. here's the standard output of the application which reports the duration of the longest task is about 3.3 minutes: 16/04/01 16:07:19 INFO TaskSetManager: Finished task 1077.0 in stage 0.0 (TID 1077) in 125425 ms on CH1SCH080051460 (1562/1563) 16/04/01 16:08:30 INFO TaskSetManager: Finished task 926.0 in stage 0.0 (TID 926) in 196776 ms on CH1SCH080100841 (1563/1563) but on spark streaming UI it's about 2.3 minutes *Summary Metrics for 1563 Completed Tasks* *Metric* *Min* *25th percentile* *Median* *75th percentile* *Max* Duration 12 s 21 s 24 s 29 s 2.3 min GC Time 0 ms 0 ms 0 ms 0 ms 62 ms Shuffle Write Size / Records 0.0 B / 0 532.9 KB / 131 1198.9 KB / 299 3.8 MB / 1161 9.0 MB / 2865 I wonder if you have any quick idea about where the missing 1 minute could be? thanks a lot, Renyi.
