On Wed, Jul 6, 2011 at 6:54 AM, Eric Payne <er...@yahoo-inc.com> wrote:
> Thanks Todd. > > Yes, the stress test is NN-only. The simulated datanodes (using > MiniDFSCluster) don't read or write actual data, only log the metadata. > > So, it sounds like the slowdown on the NN is to be expected, correct? The > race condition I was experiencing before is no longer happening, so the > benefit of correct locking has resulted in an acceptable slowdown on the > namenode. Is that correct? > How does the slowdown compare to 0.20.203 for example? We may have made the locking _too_ coarse -- ie overcompensated for the bug. -Todd > > Thanks, > -Eric > > > -----Original Message----- > > From: Todd Lipcon [mailto:t...@cloudera.com] > > Sent: Friday, July 01, 2011 7:49 PM > > To: hdfs-dev@hadoop.apache.org > > Subject: Re: HDFS on trunk is now quite slow > > > > My guess is HDFS-988 caused the slowdown by coarsening some locking that > > was > > previously incorrect. Your stress test is NN-only (metadata ops), not an > > I/O > > benchmark, right? I/O should be faster in trunk than ever before. > > > > -Todd > > > > On Fri, Jul 1, 2011 at 8:23 AM, Eric Payne <eric.payne1...@yahoo.com> > > wrote: > > > > > Hi gang, > > > > > > I ran some stress tests on the latest HDFS trunk yesterday, and the > > > performance > > > is a lot slower (sometimes 10 times slower) when compared with the HDFS > > in > > > MR-279. The HDFS in MR-279 is slightly behind trunk. The stability of > > > HDFS trunk > > > seems to be better than HDFS MR-279, but I'm not sure if the slowness > is > > > just > > > avoiding the race contitions or if they are actually fixed in trunk. > > > > > > At this point, I'm not sure what is causing this performance disparity. > > I > > > notice > > > that Block management has recently undergone significant changes in > > trunk. > > > It > > > has some new locking and it is now in its own package. Could this be > > part > > > of the > > > cause? > > > > > > Thanks, > > > -Eric > > > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > -- Todd Lipcon Software Engineer, Cloudera
