Ivan Andika created HDDS-16376:
----------------------------------
Summary: OM is stuck twice when SCM leader has a network issue
Key: HDDS-16376
URL: https://issues.apache.org/jira/browse/HDDS-16376
Project: Apache Ozone
Issue Type: Bug
Reporter: Ivan Andika
Assignee: Ivan Andika
Another incident.
h2. Problem 1: OM is stuck waiting for SCM port 9863 (block client port) with
NIC issue
The SCM leader NIC encountered an issue where it started dropping all packet.
The SCM service was able to quickly elect a new SCM leader but OM is still
stuck. It recovered for 20s after 15 minutes and then stuck again for another
30 minutes.
The first 15 minutes timeout is due to the OM waiting for the SCM call to
finish hdds.scmclient.rpc.timeout (default 15 minutes). This RPC timeout is
used in Hadoop Client.Connection.soTimeout. This soTimeout is used for the
Socket.setSoTimeout which will throw SockeTimeoutException if the socket read
and write cannot finish on time.
The main issue is that the NIC issue does not cause the RPC connection to be
broken (e.g. by TCP reset) which will trigger a fast IPC failure. The NIC issue
only causes the packet to be dropped on an active connection. Therefore, the
TCP connection keep retrying sending the packet although it's going to be
dropped.
After the 15 minutes timeout is finished, the socket timeout is hit and OM's
SCM client failover is triggered and start to the new SCM leader.
h2. Problem 2: OM is stuck again for port 9860 (container client port) after
recovering for a while
After recovering for around 20 seconds, OM is stuck again. This time is because
the SCM client for the container client port still thinks the problematic SCM
was still the leader.
Saw some logs for retying connection repeated every 20s
{code:java}
Retrying connect to server: scm1:9860. Already tried 0 time(s); maxRetries=45
Retrying connect to server: scm1:9860. Already tried 1 time(s); maxRetries=45
...
Retrying connect to server: scm1:9860. Already tried 44 time(s); maxRetries=45
{code}
The issue happened when the RPC client is trying to setup a connection to the
SCM leader but the connection cannot be established due to
ConnectTimeoutException (see Client.Connection#setupConnection).When
ConnectionTimeoutException is received, the client will wait for the total of
ipc.client.connect.timeout (20 s) × ipc.client.connect.max.retries.on.timeouts
(45) = 920s (15 minutes 20 seconds)However, this means that the SCM client
should have recovered after by 07:39 (07:24 + 15 minutes). However, it seems
that the retry cycled again and redo the whole 45 retries again.
{code:java}
08-29 07:53:49.242: Retrying connect to server: scm1:9860. Already tried 44
time(s); maxRetries=45
# cycling back to 0
08-29 07:54:29.282: Retrying connect to server: scm1:9860. Already tried 0
time(s); maxRetries=45 / code placeholder
{code}
Seems the issue is that SCM failover is not atomic, meaning that a IPC client
can trigger setupConnection while the SCM is undergoing a failover and trigger
another connection timeout retry.
h2. Possible solutions
h3. Reduce the RPC call timeout
We can check the timeout related configurations
* {{Connection timeout}}
** {{ipc.client.connect.max.retries.on.timeouts}} (default 45)
** ipc.client.connect.timeout (default 20s)
* {{RPC timeout}}
** {{hdds.scmclient.rpc.timeout}} (default 15s){{{{}}{}}}{{{{}}{}}}
h3. Make SCM client aware of other client failovers
The current issue is that both container client and block client have their own
failover states which can cause 2x timeout. The question is whether it is
possible to make one SCM client failover to notify the other SCM client to not
send a request to another request.
h3. Fix the SCM failover race
The important race is that SCM failover happens outside Hadoop IPC:
# Hadoop IPC first exhausts its 45 connect-timeout retries.
# It returns an exception to the Ozone SCM {{{}RetryProxy{}}}.
# The SCM retry policy waits 2 seconds.
# The {{RetryProxy}} then advances to the next configured SCM.
During the short interval between steps 1 and 4, another concurrent OM request
can obtain the still-current {{scm1}} proxy and create a new Hadoop IPC
connection. Its local count starts at zero and it can independently spend
another ~15 minutes retrying.
The counter is declared locally for each {{setupConnection()}} call, so it
necessarily resets for a newly created connection.
The client should switch after a completed failover, but it does not atomically
prevent already-starting or concurrent requests from beginning a fresh long
connect-retry cycle on the old node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
