Improve CheckPoint node GC performance
--------------------------------------
Key: HDFS-2555
URL: https://issues.apache.org/jira/browse/HDFS-2555
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Reporter: Bochun Bai
CheckPoint NameNode uses a lot of memory to merge the edits to the fsimage.
And the previous checkpoint data cannot easily be GCed.
It is running with jdk-sun-1.6 with different GC parameters, JProfiler
calculated heap size shown below:
For 1MB sized fsimage, -XX:-UseConcMarkSweepGC, heap size increases 10MB
every checking period.
For 1MB sized fsimage, -XX:-UseParallelGC, heap size increases and back to
10M every checking period.
For 1.2GB sized fsimage, both GC mode, heap size increases 2.5GB every
checking period.
I have two workaround, both of them are tested work fine. But I want discuss
more about the CheckPoint design.
Workaround 1: set the check period to a very long time, and restart the java
process every hour.
Pro: solves full GC problem immediately.
Con: Cannot merge edits when it increased suddenly.
Workaround 2: also reset the INode tree structure when reset the root node.
Pro: both CMS and Parallel mode works
Con: This may be a JDK-specific problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
