[
https://issues.apache.org/jira/browse/HDDS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17488:
------------------------------
Epic Link: HDDS-17321
> Waits on a stuck SCM or Recon endpoint can take all datanode state machine
> threads and delay heartbeats to the other SCMs
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17488
> URL: https://issues.apache.org/jira/browse/HDDS-17488
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Reporter: Siyao Meng
> Priority: Minor
>
> {{RunningDatanodeState.execute}} submits one wrapper per endpoint to the
> {{DatanodeStateMachine}} thread pool. The wrapper submits the endpoint task
> to the endpoint's own executor and blocks on it for up to one heartbeat
> interval. The pool has one thread per endpoint (SCMs + 1 Recon,
> {{getEndPointTaskThreadPoolSize}}), and {{StateContext.execute}} skips a
> cycle when a task is waiting in the pool queue ({{isThreadPoolAvailable}}).
> HDDS-4386 moved the endpoint tasks to their own executors so that a slow
> Recon or SCM does not take the threads used for the others. The wait for the
> task still holds a pool thread, though, and so a stuck endpoint can still
> delay the others:
> * Every cycle adds one wait on the stuck endpoint, held for up to one
> interval.
> * With regular cycles only, that wait ends about when the next cycle starts,
> so the other SCMs and Recon are still heartbeated every interval.
> * {{DatanodeStateMachine.triggerHeartbeat()}} (close container action,
> pipeline action, ICR) interrupts the cycle and starts the next one at once.
> Each such extra cycle adds another wait.
> * With as many extra cycles within one interval as there are other endpoints
> (3 with 3 SCMs and Recon), all threads are held by waits on the stuck
> endpoint. The tasks for the other SCMs wait in the queue until the oldest
> wait times out, and every cycle started meanwhile is skipped.
> So heartbeats to the healthy SCMs, including the immediate ones that carry
> close container and pipeline actions, can be up to about one heartbeat
> interval late. On its own that is not enough to mark the datanode stale. The
> stuck endpoint can be Recon as well, which is the case HDDS-4386 fixed. This
> is from code reading; not reproduced.
> Proposed fix: do not block a pool thread for the endpoint task. For example,
> keep the futures from the endpoint executors and wait for them in
> {{RunningDatanodeState.await}} with the cycle deadline, or skip an endpoint
> whose previous task is still running.
> Related: HDDS-4386, HDDS-17269 (queued tasks on the stuck endpoint),
> HDDS-17322 (why an endpoint can be stuck for about 15 minutes)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
