getMissingParentStages(stage) would be called for the stage (being re-submitted)
If there is no missing parents, submitMissingTasks() would be called. If there is missing parent(s), the parent would go through the same flow. I don't see issue in this part of the code. Cheers On Thu, Dec 24, 2015 at 5:19 AM, Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > While reviewing DAGScheduler, and where failedStages internal > collection of failed staged ready for resubmission is used, I came > across a question for which I'm looking an answer to. Any hints would > be greatly appreciated. > > When resubmitFailedStages [1] is executed, and there are any failed > stages, they are resubmitted using submitStage [2], but before it > happens, failedStages is cleared [3] so when submitStage is called > that will ultimately call submitMissingTasks for the stage, it checks > whether the stage is in failedStages (among the other sets for waiting > and running stages) [4]. > > My naive understanding is that the call to submitStage is a no-op in > this case, i.e. nothing really happens and the if expression will > silently finish without doing anything useful until some other event > happens that changes the status of the failed stages into waiting > ones. > > Is my understanding incorrect? Where? Could the call to submitStage be > superfluous? Please guide in the right direction. Thanks. > > [1] > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L734 > [2] > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L743 > [3] > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L741 > [4] > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L919 > > Pozdrawiam, > Jacek > > Jacek Laskowski | https://medium.com/@jaceklaskowski/ > Mastering Apache Spark > ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ > Follow me at https://twitter.com/jaceklaskowski > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
