ivoson commented on code in PR #52335:
URL: https://github.com/apache/spark/pull/52335#discussion_r2361649560
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1462,17 +1475,27 @@ private[spark] class DAGScheduler(
s"`${config.STAGE_MAX_CONSECUTIVE_ATTEMPTS.key}`."
abortStage(stage, reason, None)
} else {
- val missing = getMissingParentStages(stage).sortBy(_.id)
+ val missing = try {
+ getMissingParentStages(stage).sortBy(_.id)
+ } catch {
+ case e: RpcTimeoutException =>
+ val reason = s"failed to get missing parent stages for $stage " +
+ s"(name=${stage.name}) due to rpc timeout."
+ abortStage(stage, reason, Some(e))
Review Comment:
Making the stage submit process async seems like a big change for the
DAGScheduler, If we don't want to block the `DAGScheduler` I think we just
abort the stage or submit the missing parents if any even though the caches are
available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
