Xiaoqiao He created HDFS-15885:
----------------------------------
Summary: RBF: Data loss when Router setup connection timeout
Key: HDFS-15885
URL: https://issues.apache.org/jira/browse/HDFS-15885
Project: Hadoop HDFS
Issue Type: Sub-task
Components: rbf
Reporter: Xiaoqiao He
I have met one corner case which could loss data recently, it is very similar
to HDFS-15079.
Considering the following case:
A. Client send `create` RPC request to Router A at first, then Router A try to
setup new connection for this RPC request to NameNode but not setup connection
successfully in time.
B. Client failover to Router B because request timeout (60s at default IIRC).
C. Router B run normally (include RPC `create` and `complete`) and return to
Client.
D. After a while (more than 10min), Router A is back working and send `create`
to NameNode again, then this file is overwrite and data loss.
I have to state, we have replaced the ClientId and CallId of RPC with Client's
id at Router side rather that generated by Router in my deployment.
After deep dig, we found that setup connection will cost very long time when
meet some network issues. At the worst case, it will take (60 * 3 + 45 * 20) *
5 seconds (far greater than 10min - RetryCache expiry time) for setup
connections which is related with `maxRetriesOnSocketTimeouts`,
`connectionTimeout`, `maxRetriesOnSasl` and `rpcTimeout`. In this case, it
will not covered by `RetryCache` (10min by default) at NameNode side.
IMO, we should to offer the basic configuration suggestion for Router
(especially for RPC layer) to avoid Data Loss case again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
