Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37362682
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -207,6 +211,17 @@ private[yarn] class YarnAllocator(
}
/**
+ * Gets the executor loss reason for a disconnected executor.
+ * Note that this method is expected to be called exactly once per
executor ID.
+ */
+ def getExecutorLossReason(executorId: String): ExecutorLossReason =
synchronized {
+ allocateResources()
+ // Expect to be asked for a loss reason once and exactly once.
+ assert(completedExecutorExitReasons.contains(executorId))
--- End diff --
Yeah, I don't think we'll be able to get away from that. Hopefully the
"poll & eventually give up" case will be rare - basically if you get to this
point and you still don't know the reason, decrease the RM heartbeat interval
and wake up the reporter thread.
The interval is currently stored in a variable local to that thread
(`nextAllocationInterval`) but it shouldn't be hard to change that. You may
need to tweak the backoff logic to also consider whether we're waiting for the
status of a dead executor.
Regarding cleanup, maybe by using `RpcEndpointRef.ask` instead of
`askWithRetry` we can make this a little easier to follow:
- driver does `ask(GetExecutorLossReason)` with a reasonable timeout
- AM queues up that request and when the reason for that executor is known,
it replies to it.
That way, if the timeout expires, the driver will fail the executor, but
there won't be any dangling state in the AM (since the RM will eventually
report that executor's status, and the queued up message will be replied to,
except the reply will be thrown away by the driver).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
