[
https://issues.apache.org/jira/browse/HDDS-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-16376:
-------------------------------
Description:
Another incident (approx 1 hour of downtime).
The SCM leader NIC encountered an issue where it started dropping all packet.
The SCM service was able to quickly elect a new SCM leader but OM is still
stuck. It recovered for 20s after 15 minutes and then stuck again for another
30 minutes.
h2. Problem 1: OM is stuck waiting for SCM port 9863 (block client port) with
NIC issue
The first 15 minutes timeout is due to the OM waiting for the SCM call to
finish hdds.scmclient.rpc.timeout (default 15 minutes). This RPC timeout is
used in Hadoop Client.Connection.soTimeout. This soTimeout is used for the
Socket.setSoTimeout which will throw SocketTimeoutException if the socket read
and write cannot finish on time.
The main issue is that the NIC issue does not cause the RPC connection to be
broken (e.g. by TCP reset) which will trigger a fast IPC failure. The NIC issue
only causes the packet to be dropped on an active connection. Therefore, the
TCP connection keep retrying sending the packet although it's going to be
dropped.
After the 15 minutes timeout is finished, the socket timeout is hit and OM's
SCM client failover is triggered and start to the new SCM leader.
h2. Problem 2: OM is stuck again for port 9860 (container client port) after
recovering for a while
After recovering for around 20 seconds, OM is stuck again. This time is because
the SCM client for the container client port still thinks the problematic SCM
was still the leader.
Saw some logs for retying connection repeated every 20s
{code:java}
Retrying connect to server: scm1:9860. Already tried 0 time(s); maxRetries=45
Retrying connect to server: scm1:9860. Already tried 1 time(s); maxRetries=45
...
Retrying connect to server: scm1:9860. Already tried 44 time(s); maxRetries=45
{code}
The issue happened when the RPC client is trying to setup a connection to the
SCM leader but the connection cannot be established due to
ConnectTimeoutException (see Client.Connection#setupConnection).When
ConnectionTimeoutException is received, the client will wait for the total of
ipc.client.connect.timeout (20 s) × ipc.client.connect.max.retries.on.timeouts
(45) = 920s (15 minutes 20 seconds)However, this means that the SCM client
should have recovered after by 07:39 (07:24 + 15 minutes). However, it seems
that the retry cycled again and redo the whole 45 retries again.
{code:java}
08-29 07:53:49.242: Retrying connect to server: scm1:9860. Already tried 44
time(s); maxRetries=45
# cycling back to 0
08-29 07:54:29.282: Retrying connect to server: scm1:9860. Already tried 0
time(s); maxRetries=45 / code placeholder
{code}
Seems the issue is that SCM failover is not atomic, meaning that a IPC client
can trigger setupConnection while the SCM is undergoing a failover and trigger
another connection timeout retry.
h2. Problem 3: S3G does not recover by itself after full OM and SCM recovery
S3G threads are getting stuck even after OM and SCM have recovered and required
manual restart.
h2. Possible solutions
h3. Reduce the RPC call timeout
We can check the timeout related configurations
* {{Connection timeout}}
** {{ipc.client.connect.max.retries.on.timeouts}} (default 45)
** ipc.client.connect.timeout (default 20s)
* {{RPC timeout}}
** {{hdds.scmclient.rpc.timeout}} (default 15m)
Are these reasonable default in the case of OM -> SCM since RPC timeout causes
OM to be stuck?
h3. Make SCM client aware of other client failovers
The current issue is that both container client and block client have their own
failover states which can cause 2x timeout. The question is whether it is
possible to make one SCM client failover to notify the other SCM client to not
send a request to another request.
h3. Fix the SCM failover race
The important race is that SCM failover happens outside Hadoop IPC:
# Hadoop IPC first exhausts its 45 connect-timeout retries.
# It returns an exception to the Ozone SCM {{{}RetryProxy{}}}.
# The SCM retry policy waits 2 seconds.
# The {{RetryProxy}} then advances to the next configured SCM.
During the short interval between steps 1 and 4, another concurrent OM request
can obtain the still-current {{scm1}} proxy and create a new Hadoop IPC
connection. Its local count starts at zero and it can independently spend
another ~15 minutes retrying.
The counter is declared locally for each {{setupConnection()}} call, so it
necessarily resets for a newly created connection.
The client should switch after a completed failover, but it does not atomically
prevent already-starting or concurrent requests from beginning a fresh long
connect-retry cycle on the old node.
was:
Another incident (approx 1 hour of downtime).
The SCM leader NIC encountered an issue where it started dropping all packet.
The SCM service was able to quickly elect a new SCM leader but OM is still
stuck. It recovered for 20s after 15 minutes and then stuck again for another
30 minutes.
h2. Problem 1: OM is stuck waiting for SCM port 9863 (block client port) with
NIC issue
The first 15 minutes timeout is due to the OM waiting for the SCM call to
finish hdds.scmclient.rpc.timeout (default 15 minutes). This RPC timeout is
used in Hadoop Client.Connection.soTimeout. This soTimeout is used for the
Socket.setSoTimeout which will throw SocketTimeoutException if the socket read
and write cannot finish on time.
The main issue is that the NIC issue does not cause the RPC connection to be
broken (e.g. by TCP reset) which will trigger a fast IPC failure. The NIC issue
only causes the packet to be dropped on an active connection. Therefore, the
TCP connection keep retrying sending the packet although it's going to be
dropped.
After the 15 minutes timeout is finished, the socket timeout is hit and OM's
SCM client failover is triggered and start to the new SCM leader.
h2. Problem 2: OM is stuck again for port 9860 (container client port) after
recovering for a while
After recovering for around 20 seconds, OM is stuck again. This time is because
the SCM client for the container client port still thinks the problematic SCM
was still the leader.
Saw some logs for retying connection repeated every 20s
{code:java}
Retrying connect to server: scm1:9860. Already tried 0 time(s); maxRetries=45
Retrying connect to server: scm1:9860. Already tried 1 time(s); maxRetries=45
...
Retrying connect to server: scm1:9860. Already tried 44 time(s); maxRetries=45
{code}
The issue happened when the RPC client is trying to setup a connection to the
SCM leader but the connection cannot be established due to
ConnectTimeoutException (see Client.Connection#setupConnection).When
ConnectionTimeoutException is received, the client will wait for the total of
ipc.client.connect.timeout (20 s) × ipc.client.connect.max.retries.on.timeouts
(45) = 920s (15 minutes 20 seconds)However, this means that the SCM client
should have recovered after by 07:39 (07:24 + 15 minutes). However, it seems
that the retry cycled again and redo the whole 45 retries again.
{code:java}
08-29 07:53:49.242: Retrying connect to server: scm1:9860. Already tried 44
time(s); maxRetries=45
# cycling back to 0
08-29 07:54:29.282: Retrying connect to server: scm1:9860. Already tried 0
time(s); maxRetries=45 / code placeholder
{code}
Seems the issue is that SCM failover is not atomic, meaning that a IPC client
can trigger setupConnection while the SCM is undergoing a failover and trigger
another connection timeout retry.
h2. Possible solutions
h3. Reduce the RPC call timeout
We can check the timeout related configurations
* {{Connection timeout}}
** {{ipc.client.connect.max.retries.on.timeouts}} (default 45)
** ipc.client.connect.timeout (default 20s)
* {{RPC timeout}}
** {{hdds.scmclient.rpc.timeout}} (default 15m)
Are these reasonable default in the case of OM -> SCM since RPC timeout causes
OM to be stuck?
h3. Make SCM client aware of other client failovers
The current issue is that both container client and block client have their own
failover states which can cause 2x timeout. The question is whether it is
possible to make one SCM client failover to notify the other SCM client to not
send a request to another request.
h3. Fix the SCM failover race
The important race is that SCM failover happens outside Hadoop IPC:
# Hadoop IPC first exhausts its 45 connect-timeout retries.
# It returns an exception to the Ozone SCM {{{}RetryProxy{}}}.
# The SCM retry policy waits 2 seconds.
# The {{RetryProxy}} then advances to the next configured SCM.
During the short interval between steps 1 and 4, another concurrent OM request
can obtain the still-current {{scm1}} proxy and create a new Hadoop IPC
connection. Its local count starts at zero and it can independently spend
another ~15 minutes retrying.
The counter is declared locally for each {{setupConnection()}} call, so it
necessarily resets for a newly created connection.
The client should switch after a completed failover, but it does not atomically
prevent already-starting or concurrent requests from beginning a fresh long
connect-retry cycle on the old node.
> OM is stuck twice when SCM leader has a network issue
> -----------------------------------------------------
>
> Key: HDDS-16376
> URL: https://issues.apache.org/jira/browse/HDDS-16376
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Another incident (approx 1 hour of downtime).
> The SCM leader NIC encountered an issue where it started dropping all packet.
> The SCM service was able to quickly elect a new SCM leader but OM is still
> stuck. It recovered for 20s after 15 minutes and then stuck again for another
> 30 minutes.
> h2. Problem 1: OM is stuck waiting for SCM port 9863 (block client port) with
> NIC issue
> The first 15 minutes timeout is due to the OM waiting for the SCM call to
> finish hdds.scmclient.rpc.timeout (default 15 minutes). This RPC timeout is
> used in Hadoop Client.Connection.soTimeout. This soTimeout is used for the
> Socket.setSoTimeout which will throw SocketTimeoutException if the socket
> read and write cannot finish on time.
> The main issue is that the NIC issue does not cause the RPC connection to be
> broken (e.g. by TCP reset) which will trigger a fast IPC failure. The NIC
> issue only causes the packet to be dropped on an active connection.
> Therefore, the TCP connection keep retrying sending the packet although it's
> going to be dropped.
> After the 15 minutes timeout is finished, the socket timeout is hit and OM's
> SCM client failover is triggered and start to the new SCM leader.
> h2. Problem 2: OM is stuck again for port 9860 (container client port) after
> recovering for a while
> After recovering for around 20 seconds, OM is stuck again. This time is
> because the SCM client for the container client port still thinks the
> problematic SCM was still the leader.
> Saw some logs for retying connection repeated every 20s
> {code:java}
> Retrying connect to server: scm1:9860. Already tried 0 time(s); maxRetries=45
> Retrying connect to server: scm1:9860. Already tried 1 time(s); maxRetries=45
> ...
> Retrying connect to server: scm1:9860. Already tried 44 time(s);
> maxRetries=45 {code}
> The issue happened when the RPC client is trying to setup a connection to the
> SCM leader but the connection cannot be established due to
> ConnectTimeoutException (see Client.Connection#setupConnection).When
> ConnectionTimeoutException is received, the client will wait for the total of
> ipc.client.connect.timeout (20 s) ×
> ipc.client.connect.max.retries.on.timeouts (45) = 920s (15 minutes 20
> seconds)However, this means that the SCM client should have recovered after
> by 07:39 (07:24 + 15 minutes). However, it seems that the retry cycled again
> and redo the whole 45 retries again.
> {code:java}
> 08-29 07:53:49.242: Retrying connect to server: scm1:9860. Already tried 44
> time(s); maxRetries=45
> # cycling back to 0
> 08-29 07:54:29.282: Retrying connect to server: scm1:9860. Already tried 0
> time(s); maxRetries=45 / code placeholder
> {code}
> Seems the issue is that SCM failover is not atomic, meaning that a IPC client
> can trigger setupConnection while the SCM is undergoing a failover and
> trigger another connection timeout retry.
> h2. Problem 3: S3G does not recover by itself after full OM and SCM recovery
> S3G threads are getting stuck even after OM and SCM have recovered and
> required manual restart.
> h2. Possible solutions
> h3. Reduce the RPC call timeout
> We can check the timeout related configurations
> * {{Connection timeout}}
> ** {{ipc.client.connect.max.retries.on.timeouts}} (default 45)
> ** ipc.client.connect.timeout (default 20s)
> * {{RPC timeout}}
> ** {{hdds.scmclient.rpc.timeout}} (default 15m)
> Are these reasonable default in the case of OM -> SCM since RPC timeout
> causes OM to be stuck?
> h3. Make SCM client aware of other client failovers
> The current issue is that both container client and block client have their
> own failover states which can cause 2x timeout. The question is whether it is
> possible to make one SCM client failover to notify the other SCM client to
> not send a request to another request.
> h3. Fix the SCM failover race
> The important race is that SCM failover happens outside Hadoop IPC:
> # Hadoop IPC first exhausts its 45 connect-timeout retries.
> # It returns an exception to the Ozone SCM {{{}RetryProxy{}}}.
> # The SCM retry policy waits 2 seconds.
> # The {{RetryProxy}} then advances to the next configured SCM.
> During the short interval between steps 1 and 4, another concurrent OM
> request can obtain the still-current {{scm1}} proxy and create a new Hadoop
> IPC connection. Its local count starts at zero and it can independently spend
> another ~15 minutes retrying.
> The counter is declared locally for each {{setupConnection()}} call, so it
> necessarily resets for a newly created connection.
> The client should switch after a completed failover, but it does not
> atomically prevent already-starting or concurrent requests from beginning a
> fresh long connect-retry cycle on the old node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
