Marton Elek created HDDS-2542: --------------------------------- Summary: Race condition between read and write stateMachineData Key: HDDS-2542 URL: https://issues.apache.org/jira/browse/HDDS-2542 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Marton Elek
The write payload (the chunk itself) is sent to the Ratis as an external, binary byte array. It's not part of the LogEntry and saved from an async thread with calling ContainerStateMachine.writeStateMachineData As it's an async thread it's possible that the stateMachineData is not yet written when the data should be sent to the followers in the next heartbeat. By design a cache is used to avoid this issue but there are multiple problems with the cache. First, the current cache size is chunkExecutor.getCorePoolSize() which is not enough. By default it means 10 executor thread and a cache with size 10. But in case of one very slow and nine very fast writer the cache entries can be invalidated before the write. In my tests (freon datanode-chunk-writer-generator) I have seen missed cache hits even with cache size 5000. Second: as the readStateMachineData and writeStateMachien data are called from two different thread there is a race condition independent from the the cache size. It's possible that the write thread has not yet added the data to the cache but the read thread needs it. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org