Roman, what is your model for how test results from Bigtop should feed back into Hadoop-2 development? With the understanding that (a) software does have bugs, and (b) you're not going to get an SLA on community-sponsored software, what are your ideas for how to close the loop better?
Would "CI" runs of Bigtop against branch-2 be feasible, as Arun suggests? How should we accomodate changes in individual components (Hadoop Core, but others as well) that may require changes in one or more other components? How does Bigtop keep doing a viable nightly build in that chaotic environment? Is this a previously solved problem? Thanks, --Matt On Wed, May 15, 2013 at 4:47 PM, Arun C Murthy <a...@hortonworks.com> wrote: > > On May 15, 2013, at 3:50 PM, Roman Shaposhnik wrote: > > > Arun, > > > > am I reading yours answer to my binary question correctly? It is a 'no'. > > No. > > > > > My reading of your response is that while you appreciate the feedback > > Bigtop is providing you're not of an opinion that investigating the level > > of stability of Hadoop wrt. downstream any further than what is currently > > happening would be a worthy investment of Hadoop's community > > (or your personal for that matter) time? > > Everyone is welcome to contribute in any and all manner. I can't speak for > everyone. > It would be useful if Bigtop could run regressions on releases here > consistently. We've also talked in the past about running Bigtop on > branch-2, nightly. Is that something you could help with? You'd earn my > personal gratitude. > > thanks, > Arun > > > > > Thanks, > > Roman. > > > > On Wed, May 15, 2013 at 3:02 PM, Arun C Murthy <a...@hortonworks.com> > wrote: > >> Roman, > >> > >> Furthermore, before we rush into finding flaws and scaring kids at > night it would be useful to remember one thing: > >> Software has *bugs*. We can't block any release till the entire > universe validates it, in fact they won't validate it if we don't release > since are at the bottom of the stack. > >> > >> Any help prior to the release is welcome; I know people who work for > the same employer as I do have plans to do further testing after we freeze > apis via the beta release(s). I hope and pray others can join this effort - > thanks to everyone who already has. > >> > >> Again, freezing APIs and protocols is the primary aim of 2.0.5-beta. > There are no guarantees it's 100% bug-free, we can never make such > guarantees anyway. > >> > >> If, and when, we find bugs with 2.0.5-beta I'm more than happy to > quickly turn around and make more releases (2.0.6-beta, 2.0.7-beta). > Obviously I'll make a call on which bugs are critical - feedback to help me > decide is, as always, welcome. > >> I've been clear, many times, that we might need more than one beta > release to iron out bugs etc. > >> > >> None of this should be a surprise - this has happened many, many times > in the lifetime of this and other projects. 2.0.3-alpha vis-a-vis > 2.0.4-alpha is the most recent example - it won't be the last. > >> > >> So, I hope, concludes this meme. > >> > >> thanks, > >> Arun > >> > >> On May 15, 2013, at 2:20 PM, Arun C Murthy wrote: > >> > >>> Great summary, thanks Vinod. > >>> > >>> On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote: > >>> > >>>> > >>>> Roman, I keep this same argument again and again. Should've refuted > earlier. > >>>> > >>>> Please list down all the issues that BigTop ran into *because of* new > features. You continue to argue that new features are destabilizing 2.0.*, > which I don't agree with at all. 2.0.3-alpha was the last time major > features got merged in, and we found blockers irrespective of those. > >>>> > >>>> MAPREDUCE-5240 specifically isn't due to any feature merge. This was > a bug. I'd say this is a long standing bug in 2.0.x. You sure this passed > in 2.0.3? Even so, this is mostly broken by another bug-fix and *not* > because of any feature. > >>>> > >>>> I quickly checked other bugs you reported in 2.0.x: > >>>> - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was > again a long standing issue in 2.0.x > >>>> - MAPREDUCE-3728 is similar > >>>> - MAPREDUCE-5117 is similar > >>>> - MAPREDUCE-4219 was a security related feature request from you. > >>>> - MAPREDUCE-3916 was because of new proxy-server added. > >>>> > >>>> I am not arguing that new features *may* destabilize the branch, but > you've repeatedly stated this as if that were a fact. > >>>> > >>>> Really appreciate the testing done by BigTop, but please don't > distort the facts. > >>>> > >>>> Thanks, > >>>> +Vinod > >>>> > >>>> > >>>> On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote: > >>>> > >>>>> Please tell me if my expectations are incorrect, but to me the -beta > would > >>>>> signify it being a 'safe' target for the downstream components. > We're still > >>>>> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is > >>>>> a good example) that essentially mean DOA for downstream that depends > >>>>> on this functionality. > >>>>> > >>>>> Are we comfortable with delivering 2.0.5-beta and later on starting > >>>>> to discover things like MAPREDUCE-5240 more or less accidentally? > >>>> > >>> > >>> -- > >>> Arun C. Murthy > >>> Hortonworks Inc. > >>> http://hortonworks.com/ > >>> > >>> > >> > >> -- > >> Arun C. Murthy > >> Hortonworks Inc. > >> http://hortonworks.com/ > >> > >> > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > >
