Elek, Marton created HDDS-1785:
----------------------------------
Summary: OOM error in Freon due to the concurrency handling
Key: HDDS-1785
URL: https://issues.apache.org/jira/browse/HDDS-1785
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Elek, Marton
HDDS-1530 modified the concurrent framework usage of Freon (RandomKeyGenerator).
The new approach uses separated tasks (Runnable) to create the
volumes/buckets/keys.
Unfortunately it doesn't work very well in some cases.
# When Freon starts it creates an executor with fixed number of threads (10)
# The first loop submits numOfVolumes (10) VolumeProcessor tasks to the
executor
# The 10 threads starts to execute the 10 VolumeProcessor tasks
# Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor
tasks. All together 10000 tasks are submitted to the executor.
# The 10 threads starts to execute the first 10 BucketProcessor tasks, they
starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
# At this point of the time no keys are generated, but the next 10
BucketProcessor tasks are started to execute..
# To execute the first key creation we should process all the BucketProcessor
tasks which means that all the Key creation tasks (10 * 1000 * 500 000) are
created and added to the executor
# Which requires a huge amount of time and memory
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
