xudongcao created HDFS-14484:
--------------------------------
Summary: Ozone: Freon: Support big files larger than 2GB and fix
OOM.
Key: HDFS-14484
URL: https://issues.apache.org/jira/browse/HDFS-14484
Project: Hadoop HDFS
Issue Type: Bug
Components: benchmarks
Reporter: xudongcao
Assignee: xudongcao
*Current problems:*
1. Freon does not support big files larger than 2GB because it use a int type
for keysize.
2. Freon allocates the entire key buffer at once (this is exactly the reson
why it cannot support files larger than 2GB, because the maximum size of a java
array byte[] is 2G), if the test pressure is big enough, freon will report OOM
exception frequently.
*And we need:*
1. Make sure freon can support big files larger than 2GB.
2. Use an internal small buffer repeatedly than allocating the entire key-size
buffer at once, the typical size of the small buffer is 4K and it can be
configured.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
