Marton Elek created HDDS-2542:
---------------------------------

             Summary: Race condition between read and write stateMachineData
                 Key: HDDS-2542
                 URL: https://issues.apache.org/jira/browse/HDDS-2542
             Project: Hadoop Distributed Data Store
          Issue Type: Bug
          Components: Ozone Datanode
            Reporter: Marton Elek


The write payload (the chunk itself) is sent to the Ratis as an external, 
binary byte array. It's not part of the LogEntry and saved from an async thread 
with calling ContainerStateMachine.writeStateMachineData

 

As it's an async thread it's possible that the stateMachineData is not yet 
written when the data should be sent to the followers in the next heartbeat.

By design a cache is used to avoid this issue but there are multiple problems 
with the cache.

First, the current cache size is chunkExecutor.getCorePoolSize() which is not 
enough. By default it means 10 executor thread and a cache with size 10. But in 
case of one very slow and nine very fast writer the cache entries can be 
invalidated before the write.

In my tests (freon datanode-chunk-writer-generator) I have seen missed cache 
hits even with cache size 5000.

Second: as the readStateMachineData and writeStateMachien data are called from 
two different thread there is a race condition independent from the the cache 
size. It's possible that the write thread has not yet added the data to the 
cache but the read thread needs it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Reply via email to