Re: Review request: trunk->HDFS-1623 merge

2012-02-09 Thread Todd Lipcon
Thanks Suresh. I committed the merge to the branch. -Todd On Thu, Feb 9, 2012 at 4:32 PM, Suresh Srinivas wrote: > I looked at the merge. It looks good. +1. > > On Wed, Feb 8, 2012 at 9:08 PM, Todd Lipcon wrote: > >> The branch developed some new conflicts due to recent changes in trunk >> affe

Re: Review request: trunk->HDFS-1623 merge

2012-02-09 Thread Suresh Srinivas
I looked at the merge. It looks good. +1. On Wed, Feb 8, 2012 at 9:08 PM, Todd Lipcon wrote: > The branch developed some new conflicts due to recent changes in trunk > affecting the RPC between the DN and the NN (the "StorageReport" > stuff). I've done a new merge to address these conflicts here

Re: Review request: trunk->HDFS-1623 merge

2012-02-09 Thread Todd Lipcon
Updated the merge against current trunk and HA branch. Here's the github link: https://github.com/toddlipcon/hadoop-common/commits/ha-merge-20120209 And the relevant diff reproduced below. If this looks mostly good, please +1 - we can continue to make improvements on the branch, but redoing the m

Re: Review request: trunk->HDFS-1623 merge

2012-02-09 Thread Todd Lipcon
On Thu, Feb 9, 2012 at 12:54 PM, Konstantin Shvachko wrote: > Ok I misunderstood the current direction of the merge. > > On the review request: > >> we don't deal with the case where OP_ADD >> contains blocks on a new file -- this is a case that doesn't happen on >> real clusters, but currently ha

Re: Review request: trunk->HDFS-1623 merge

2012-02-09 Thread Konstantin Shvachko
Ok I misunderstood the current direction of the merge. On the review request: > we don't deal with the case where OP_ADD > contains blocks on a new file -- this is a case that doesn't happen on > real clusters, but currently happens with synthetic logs generated > from the CreateEditLogs tool. I

Re: Review request: trunk->HDFS-1623 merge

2012-02-09 Thread Todd Lipcon
Hi Konstantin, To be clear, this review request is a merge from the trunk branch into the HA branch (NOT a merge INTO trunk) - we've been doing these pretty much daily since the project began, so that we track trunk closely. The idea is so that we don't have unexpected integration issues when we d

Re: Review request: trunk->HDFS-1623 merge

2012-02-09 Thread Konstantin Shvachko
I was wondering 1. What was the test plan that has been executed for testing this implementation of HA? Besides unit tests. 2. Have you done any benchmarks, comparing current cluster performance against the branch. Would be good to have numbers for both cases with HA off and HA on. I'll post these

Re: Review request: trunk->HDFS-1623 merge

2012-02-08 Thread Todd Lipcon
The branch developed some new conflicts due to recent changes in trunk affecting the RPC between the DN and the NN (the "StorageReport" stuff). I've done a new merge to address these conflicts here: https://github.com/toddlipcon/hadoop-common/tree/ha-merge-20120208 I've also addressed Aaron's com

Re: Review request: trunk->HDFS-1623 merge

2012-02-03 Thread Aaron T. Myers
On Fri, Feb 3, 2012 at 6:45 PM, Suresh Srinivas wrote: > Todd, can you please hold off for the merge till Tuesday, until some of the > other folks working on HA could catch up with some of the recent changes. > Note that this merge doesn't have anything to do with any *recent* changes to the HA b

Re: Review request: trunk->HDFS-1623 merge

2012-02-03 Thread Suresh Srinivas
Todd, can you please hold off for the merge till Tuesday, until some of the other folks working on HA could catch up with some of the recent changes. On Fri, Feb 3, 2012 at 2:10 PM, Todd Lipcon wrote: > I've got a merge pending of trunk into HDFS-1623 -- it was a bit > complicated so wanted to a

Re: Review request: trunk->HDFS-1623 merge

2012-02-03 Thread Aaron T. Myers
Hey Todd, The merge largely looks good. I agree with the general approach you took. A few small comments: 1. There's a comment in the OP_ADD case blockabout handling OP_CLOSE. This makes sense in 0.22/0.23/0.24, but in the HA branch the OP_ADD and OP_CLOSE cases are completely separate case block

Review request: trunk->HDFS-1623 merge

2012-02-03 Thread Todd Lipcon
I've got a merge pending of trunk into HDFS-1623 -- it was a bit complicated so wanted to ask for another set of eyes: https://github.com/toddlipcon/hadoop-common/tree/ha-merge-20120203 (using github since it's hard to review a merge patch via JIRA) The interesting bit of the merge was to deal wit