stragglers? Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi>
On Tue, Jul 1, 2014 at 12:40 AM, Yana Kadiyska <yana.kadiy...@gmail.com> wrote: > Hi community, this one should be an easy one: > > I have left spark.task.maxFailures to it's default (which should be > 4). I see a job that shows the following statistics for Tasks: > Succeeded/Total > > 7109/819 (1 failed) > > So there were 819 tasks to start with. I have 2 executors in that > cluster. From Spark docs it says spark.task.maxFailures is the number > of times to try a task before a job is given up. So I was imagining > that 819*4 (i.e. 3276) would be the max number to ever see in the > succeeded (accounting for retries on every possibly task). even that > 3276*2 (6552, if it's per task per executor) does not account for 7109 > successfull tasks. > > Could anyone help explain why I'm seeing such high number of succeeded > tasks? >
