Namenode in trunk has much slower performance than Namenode in MR-279 branch
----------------------------------------------------------------------------
Key: HDFS-2142
URL: https://issues.apache.org/jira/browse/HDFS-2142
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.23.0
Reporter: Eric Payne
I am measureing the performance of the namenode by running the
org.apache.hadoop.fs.loadGenerator.LoadGenerator application. This application
shows there is a very large slowdown in the processing of opens, writes,
closes, and operations per second in the trunk when compared to the MR-279
branch
There have been some race conditions and locking issues fixed in trunk, which
is a very good thing because these race conditions were causing the namenode to
crash under load conditions (see HDFS:1257). However, the slowdown to the
namenode is considerable.
I am still trying to verify which changes caused the slowdown. It was
originally suggested that the HDFS:988 may have caused the slowdown, but I
don't think it was the culprit. I have checked out and built from SVN 3
revisions previous to HDFS988 and they all have about the same performance.
Here is my environment:
Host0: namenode daemon
Host1-9: simulate many datanodes using org.apache.hadoop.hdfs.DataNodeCluster
LoadGenerator output on MR-279 branch:
Average open execution time: 1.8496516782773909ms
Average deletion execution time: 2.956340167046317ms
Average create execution time: 3.725259427992913ms
Average write_close execution time: 11.151860288534548ms
Average operations per second: 1053.3666666666666ops/s
LoadGenerator output on trunk:
Average open execution time: 28.603515625ms
Average deletion execution time: 32.20792079207921ms
Average create execution time: 32.37326732673267ms
Average write_close execution time: 82.84752475247525ms
Average operations per second: 135.13333333333333ops/s
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
