attilapiros commented on code in PR #50630:
URL: https://github.com/apache/spark/pull/50630#discussion_r2069716779
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -2138,7 +2143,8 @@ private[spark] class DAGScheduler(
// guaranteed to be determinate, so the input data of the
reducers will not change
// even if the map tasks are re-tried.
if (mapStage.isIndeterminate) {
- val rollingBackStages =
abortIndeterminateStageChildren(mapStage)._2
+ val stagesToRollback = collectSucceedingStages(mapStage)
+ val rollingBackStages = rollBackStages(stagesToRollback)
Review Comment:
I have refactored `abortIndeterminateStageChildren` into two separate
methods:
- collectSucceedingStages
- rollBackStages
This way we do not need returning a tuple anymore and we are not calculating
the "All jobs with map stage have been aborted" when it is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
