[
https://issues.apache.org/jira/browse/YUNIKORN-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109058#comment-18109058
]
Aditya Maheshwari commented on YUNIKORN-3426:
---------------------------------------------
{{GetTaskState}} - it takes read lock on FSM and should not be blocked.
GetTaskPod - It takes read lock and should not be blocked.
SetTaskPod - It is desired to avoid updating the task until its bind activities
are over. This is intentional.
We do not want to write to the task during event handling while binding is
ongoing. This is intentional.
> Pod bind retries with backoff run under the task write lock
> -----------------------------------------------------------
>
> Key: YUNIKORN-3426
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3426
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Assignee: Aditya Maheshwari
> Priority: Major
>
> Since YUNIKORN-2884 the bind goroutine in {{Task.postTaskAllocated}} takes
> {{task.lock.Lock()}} and then runs both the volume bind and the pod bind
> through {{retry.OnError(retryBackoff, ...)}}, releasing the lock only
> afterwards. {{retryBackoff}} is eight steps starting at one second, doubling,
> capped at 30 s; the file's own comment puts the total at 121 seconds.
> So while the API server is slow or failing, the task write lock is held for
> up to two minutes of sleeping plus the round trips, and everything else that
> touches that task waits behind it: the informer's pod updates
> ({{SetTaskPod}}), {{GetTaskState}} / {{GetTaskPod}} from the dispatcher and
> the scheduling loop, task events through {{task.handle}}, and the
> application-level walks that take task locks. With many tasks binding at once
> during an API-server outage the shim's per-task paths stall broadly. Not
> reproduced; the mechanism is unconditional.
> This landed after the lock annotations were written and was reported by the
> analyser when the branch was rebased; it is the kind of change the check is
> meant to catch at review.
> Fix: snapshot what the bind needs under the lock, release it, run the retry
> loop unlocked, and re-take the lock only to apply the result and drive the
> state transition, the same shape the code already uses for the pod/alias/node
> snapshot it passes into the goroutine.
> Marker: the two {{retry.OnError}} calls in {{task.go}} carry
> {{+lockblockingignore}} for this JIRA; the fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
