adoroszlai opened a new pull request #53: HDDS-2330. Random key generator can get stuck URL: https://github.com/apache/hadoop-ozone/pull/53 ## What changes were proposed in this pull request? Fix the problem that any exception/error not caught by `ObjectCreator` ends the object creation task, but Freon's main thread continues waiting indefinitely, since the exception is not stored. https://issues.apache.org/jira/browse/HDDS-2330 ## How was this patch tested? Verified that OOME is caught, reported, and results in Freon exiting with failure. ``` $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone $ docker-compose up -d $ docker-compose exec scm ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1 --replicationType RATIS --factor ONE --keySize $(echo '2^20' | bc -lq) --numOfKeys $(echo '5 * 2^10' | bc -lq) --bufferSize $(echo '2^16' | bc -lq) ... 6.66% |??????? | 341/5120 Time: 0:00:17 [pool-2-thread-1] ERROR - Exception while adding key: key-357-74353 in bucket: bucket-0-90611 of volume: vol-0-95721. java.lang.OutOfMemoryError: Java heap space at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61) at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348) at org.apache.hadoop.hdds.scm.storage.BufferPool.allocateBufferIfNeeded(BufferPool.java:81) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:233) at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:129) at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:208) at org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:190) at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:710) at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:88) at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:615) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 100.00% |?????????????????????????????????????????????????????????????????????????????????????????????????????| 5120/5120 Time: 0:00:20 java.lang.OutOfMemoryError: Java heap space *************************************************** Status: Failed Git Base Revision: e97acb3bd8f3befd27418996fa5d4b50bf2e17bf Number of Volumes created: 1 Number of Buckets created: 1 Number of Keys added: 357 Ratis replication factor: ONE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,190 Average Time spent in bucket creation: 00:00:00,030 Average Time spent in key creation: 00:00:02,826 Average Time spent in key write: 00:00:14,607 Total bytes written: 374341632 Total Execution time: 00:00:21,593 *************************************************** ``` Also verified that successful execution is not affected: ``` $ docker-compose exec scm ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1 --replicationType RATIS --factor ONE --keySize $(echo '2^20' | bc -lq) --numOfKeys 3 --bufferSize $(echo '2^16' | bc -lq) ... 100.00% |?????????????????????????????????????????????????????????????????????????????????????????????????????| 3/3 Time: 0:00:02 *************************************************** Status: Success Git Base Revision: e97acb3bd8f3befd27418996fa5d4b50bf2e17bf Number of Volumes created: 1 Number of Buckets created: 1 Number of Keys added: 3 Ratis replication factor: ONE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,083 Average Time spent in bucket creation: 00:00:00,012 Average Time spent in key creation: 00:00:00,069 Average Time spent in key write: 00:00:01,611 Total bytes written: 3145728 Total Execution time: 00:00:05,986 *************************************************** ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
