Amir Langer created HDFS-7244:
---------------------------------
Summary: Reduce Namenode memory using Flyweight pattern
Key: HDFS-7244
URL: https://issues.apache.org/jira/browse/HDFS-7244
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Amir Langer
Using the flyweight pattern can dramatically reduce memory usage in the
Namenode. The pattern also abstracts the actual storage type and allows the
decision of whether it is off-heap or not and what is the serialisation
mechanism to be configured per deployment.
The idea is to move all BlockInfo data (as a first step) to this storage using
the Flyweight pattern. The cost to doing it will be in higher latency when
accessing/modifying a block. The idea is that this will be offset with a
reduction in memory and in the case of off-heap, a dramatic reduction in memory
(effectively, memory used for BlockInfo would reduce to a very small constant
value).
This reduction will also have an huge impact on the latency as GC pauses will
be reduced considerably and may even end up with better latency results than
the original code.
I wrote a stand-alone project as a proof of concept, to show the pattern, the
data structure we can use and what will be the performance costs of this
approach.
see [Slab|https://github.com/langera/slab]
and [Slab performance
results|https://github.com/langera/slab/wiki/Performance-Results].
Slab abstracts the storage, gives several storage implementations and
implements the flyweight pattern for the application (Namenode in our case).
The stages to incorporate Slab into the Namenode is outlined in the sub-tasks
JIRAs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
