On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas <sur...@hortonworks.com>wrote:
> Todd, > > As I indicated in my comments on the jira, I think some of the design > discussions and further simplification of design should happen before the > merge. See - > > https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680 > > I still haven't heard any actual concrete suggestion for a design improvement. Just abstract notions that "it should be simpler". I could say the same about several other features that have been done in the past (e.g federation or YARN) but chose not to because I didn't participate in the development. I generally have faith that, if several other people spent several months working on a project, then there must be good reasons for the complexity that I'm probably overlooking. Several of us (not just I) spent lots of time on this over the last several months, with all of the work going on in the open. My issue with this discussion happening now is that no one saw fit to raise these questions months ago when the design was first proposed. For example, this most recent question about whether NewEpoch and PrepareRecovery should be separate RPCs could have been raised in late June when the code in question was first introduced as a patch. Nor was anything raised when I gave a "heads up" that the branch was complete and nearly ready for merge ~3 weeks ago. Only once I actually called a merge vote has this discussion started. That doesn't seem like a healthy way to do development. > I understand that you have spent a lot of time working on this. You have > indicated that you do not want to make any further design improvements. I > am willing to help by doing any improvements that comes out of the > discussions on the jira in 3077 branch and keep it up to date. I am also > willing to merge this branch into trunk. So my vote is to hold off the > merge until the discussions complete. > What are the criteria, then, for merging? I don't think it's possible to definitively "prove" that a design is "simple". At some level it's a matter of taste. So if the current design works, and is tested, and has people signed up to support it and run it, why not merge? Just like any other part of HDFS, we can continue to _improve_ it after it is in trunk. There are many features that we commit and then improve later on when someone comes up with a way to simplify it. To be clear about the current status of the vote: you are officially vetoing the merge? Thanks -Todd On Mon, Oct 8, 2012 at 5:46 PM, Todd Lipcon <t...@cloudera.com> wrote: > > > Hi Sanjay, > > > > The 7 extra days you requested beyond the original 7-day merge vote have > > now elapsed, and we have the requisite three binding +1s to merge. > > > > I'll plan to merge this late tonight unless there are any vetoes in the > > meantime. > > > > Of course we can continue to discuss the design and improve the clarity > of > > the documentation after it's in trunk, and if there's some kind of bug > I'll > > treat it as highest priority even after the merge. > > > > Thanks > > -Todd > > > > > > On Mon, Oct 1, 2012 at 10:55 AM, sanjay Radia <san...@hortonworks.com > > >wrote: > > > > > Todd, > > > Even though this work was under development over a period of time, > > > during its development it was not clear when the design was fairly > > stable > > > to begin a thorough review. Hence the time of merge is when the real > > review > > > happens in such large projects. > > > > > > I have already indicated on the jira that i do not have any > philosophical > > > objection to this work being in HDFS - hence this should not be a worry > > on > > > your part. > > > > > > The extra week will result in a more through review (hopefully this > will > > > have a side effect of perhaps easing Konstanine's concern about > > > HDFS adding such complex code). > > > > > > Lets plan to do the merge next monday. > > > > > > thanks > > > > > > sanjay > > > > > > > > > > > > > > > On Sep 28, 2012, at 3:02 PM, Todd Lipcon wrote: > > > > > > > Hey Sanjay, > > > > > > > > While I understand it's substantial and complex code, the code and > the > > > > design doc have been available for several months, and the community > > > > has certainly been aware of its development. I also gave a heads up > > > > last week that I would call a merge this week. So I feel like there > > > > has been sufficient time for interested parties to review. > > > > > > > > That said, since I was sick for much of this week and not immediately > > > > responsive to some of the questions from you and Suresh, I'm happy to > > > > agree to postpone the merge to early next week. Let's extend the vote > > > > to last until Monday end of day PST. > > > > > > > > Of course if there are follow-up questions or bugs found after the > > > > merge, you've all got my phone number and I'm not going anywhere! ;-) > > > > > > > > Thanks > > > > -Todd > > > > > > > > On Fri, Sep 28, 2012 at 12:06 PM, sanjay Radia < > san...@hortonworks.com > > > > > > wrote: > > > >> Suresh and I are still reviewing this design and patch. > > > >> The 3077 code along with the code pulled from 3092 is fairly > > > substrantial. The design is also fairly complex and involved. > > > >> I would request that we postpone the merge for another week to give > > > folks time to review this fully. > > > >> > > > >> > > > >> sanjay > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On Sep 25, 2012, at 4:02 PM, Todd Lipcon wrote: > > > >> > > > >>> Dear fellow HDFS developers, > > > >>> > > > >>> Per my email thread last week ("Heads up: merge for QJM branch > soon" > > > >>> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to > > > >>> propose merging the HDFS-3077 branch into trunk. The branch has > been > > > >>> active since mid July and has stabilized significantly over the > last > > > >>> two months. It has passed the full test suite, findbugs, and > release > > > >>> audit, and I think it's ready to merge at this point. > > > >>> > > > >>> The branch has been fully developed using the standard > > > >>> 'review-then-commit' (RTC) policy, and the design is described in > > > >>> detail in a document attached to HDFS-3077 itself. The code itself > > has > > > >>> been contributed by me, Aaron, and Eli, but I'd be remiss not to > also > > > >>> acknowledge the contributions to the design from discussions with > > > >>> Suresh, Sanjay, Henry Robinson, Patrick Hunt, Ivan Kelly, Andrew > > > >>> Purtell, Flavio Junqueira, Ben Reed, Nicholas, Bikas, Brandon, and > > > >>> others. Additionally, special thanks to Andrew Purtell and Stephen > > Chu > > > >>> for their help with cluster testing. > > > >>> > > > >>> This initial VOTE is to merge only into trunk, but, following the > > > >>> pattern of automatic failover, I expect to merge it into branch-2 > > > >>> within a few weeks as well. The merge to branch-2 should be clean, > as > > > >>> both I and Andrew Purtell have been testing on branch-2-derived > > > >>> codebases in addition to trunk. > > > >>> > > > >>> Please cast your vote by EOD Friday 9/29. Given that the branch has > > > >>> only had small changes in the last few weeks, and there was a > "heads > > > >>> up" last week, I trust this should be enough time for committers to > > > >>> cast their votes. Per our by-laws, we need a minimum of three > binding > > > >>> +1 votes from committers. > > > >>> > > > >>> I will start the voting with my own +1. > > > >>> > > > >>> Thanks > > > >>> -Todd > > > >>> -- > > > >>> Todd Lipcon > > > >>> Software Engineer, Cloudera > > > >> > > > > > > > > > > > > > > > > -- > > > > Todd Lipcon > > > > Software Engineer, Cloudera > > > > > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > > > > > -- > http://hortonworks.com/download/ > -- Todd Lipcon Software Engineer, Cloudera
