I think a distributed Jenkins setup would be great. If we had really awesome test coverage, I would be less frightened of last-minute checkins, as well. :)
On Tue, Sep 24, 2013 at 3:17 PM, Daan Hoogland <daan.hoogl...@gmail.com>wrote: > Mike, rest assured you and Marcus are not the only ones. More guarantee on > a stable master is a general concern. Personally I don't feel we need more > control on what is in the next release, if we make unit tests and automated > integration tests a priority. That is kind of a claim I do have 'the' > solution, though not well cooked ;) It's going to take a while (a colleague > said four or five releases) before we have a good enough test set and a > smoothly running continuous integration test engine. I think we at least > need the distributed Jenkins setup where you can run your own integration > tests to make sure your invested logic remains intact. This of course being > only part of 'all the' answers. > > regards, > > > On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > > > I was a bit hesitant to keep pushing this because there doesn't seem to > be > > a lot of support for it, but - as Marcus pointed out - I was quite > alarmed > > by the number and criticality of bugs checked in right before we cut our > > first RC for 4.2. We simply were not ready. > > > > To me, it felt like something one might do before one gets out a decent > > beta release. > > > > I certainly don't claim to have all the answers for this, but I do think > we > > need to develop some kind of a process whereby very few changes are made > > immediately prior (like a month) to the first cut of a RC. We might even > > need to discuss such changes as a community before they get checked in > > (after a certain point). > > > > As far as master not always being usable, this is a serious problem, as > > well. > > > > For example, I've been having trouble getting KVM to work and - in the > > meanwhile - my code has fallen out of date with master over the past week > > or so. However, I'm always afraid if I update from master while in the > > middle of solving one problem that I'll have more problems to deal with > > before I can get back to the initial problem (because something didn't > work > > in master). > > > > Again, I don't claim to have any solution for this problem, but I am > happy > > to help brainstorm. > > > > > > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <shadow...@gmail.com > > >wrote: > > > > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi > > > <animesh.chaturv...@citrix.com> wrote: > > > > > > > > > > > >> -----Original Message----- > > > >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > > > >> Sent: Monday, September 23, 2013 12:25 PM > > > >> To: dev@cloudstack.apache.org > > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or > > revamp > > > >> release process > > > >> > > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi" > > > >> <animesh.chaturv...@citrix.com> > > > >> wrote: > > > >> > > > > >> > > > > >> > > > > >> > > -----Original Message----- > > > >> > > From: Marcus Sorensen [mailto:shadow...@gmail.com] > > > >> > > Sent: Monday, September 23, 2013 11:38 AM > > > >> > > To: dev@cloudstack.apache.org > > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or > > revamp > > > >> > > release process > > > >> > > > > > >> > > Guys, I think we are not currently in a state to handle > > time-based > > > >> > > releases. Until we can cut master at any time and have it > > > >> > > releasable, or at least at a reasonable RC-level matching > minimum > > > >> > > tested requirements, it's just going to continue to be an > exercise > > > >> > > in frustration to cut RCs simply because we hit a deadline. > > > >> > [Animesh>] David is going to propose Release Criterion up for > > > >> > discussion > > > >> as per his thread [1] > > > >> > > > >> I see that thread more about defining what minimum bar we should > > always > > > >> have master at in order to meet time-based releases. Its where we > want > > > >> to go, but not what to do in the meantime. > > > > [Animesh>] His proposal is not just for master, but also for deciding > > > the release exit criterion and IMO is something we should follow for > > 4.3.0 > > > and onwards > > > > > > Yes, I know. What I meant was that it will be a step toward > > > stabilizing master, until we do that I'm not convinced we can adhere > > > to any time-based expectation). It still doesn't fix our issue if > > > we're going to insist on time-based releases, it just (from my > > > undertanding) sets a bar for what is acceptable and what isn't, for > > > any release. It stops the argument of "should we release with this > > > bug". > > > > > > >> > > > >> > > > > > >> > > Maybe we can get away with sticking to time-based if we revamp > our > > > >> > > schedule and procedures, I don't know, but in light of how 4.1 > > > >> > > (dragged on so long that some were seriously considering > > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six > > rounds > > > >> > > of votes so > > > >> > > far) have worked it's probably worth discussing. > > > >> > > > > > >> > > Any suggestions on what might be better? It's been mentioned in > > the > > > >> > > past that it's a chicken-egg thing, many really don't try it > until > > > >> > > we hit an RC, which causes multiple iterations. I do agree that > > many > > > >> > > don't take it seriously until we start cutting artifacts, but > > maybe > > > >> > > we do this in a more deliberate fashion instead of jumping right > > to > > > >> > > the vote. After feature/code freeze, cut some alpha artifacts, > > wait > > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at some > > > >> > > point anyone can propose that certain artifacts (or a new set of > > > >> > > artifacts) be put up for a vote as an RC. This gives us a way to > > > >> > > signal that we're gearing up for release and gives plenty of > time > > > >> > > for people to test their components, or see the [PROPOSAL] and > say > > > >> > > 'oh crap, I had better test my stuff', prior to cutting an RC. > > > >> > > Maybe this wouldn't help in practice, but I think right now we > go > > > >> > > from telling the community "code is frozen, don't check anything > > in > > > >> > > unless its a bug fix" to "here's our RC, try it out", without a > > > >> formal testing window. > > > >> > > I realize the whole thing should be a testing window, but I > don't > > > >> > > think it's conveyed well. > > > >> > > > > >> > [Animesh>] After the code freeze is all the stabilization and > > > >> > integration > > > >> testing phase and has been documented at [2]. No one should be > > waiting > > > >> until the RC to test their components for the first time. It should > be > > > >> happening after code freeze. > > > >> > > > >> > > > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm > > > >> > [2] > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases > > > >> > > > > >> > > > >> Got it. As mentioned I realize that the whole time there is supposed > > to > > > >> be testing, but its not really working that way in practice. People > > are > > > >> volunteers, they forget where things are, or they dont want to mess > > with > > > >> it unless there is an indication that its semi-stable, and then > > suddenly > > > >> an RC is thrown over the fence and we go through iterations of RC. > By > > > >> the time the RC comes through we should be done testing and just > > verify > > > >> that someone's last minute bug fix didn't cause a regression or > > > >> something. > > > > [Animesh>] RC is not thrown in it is discussed as part of the release > > > schedule. After the code-freeze date everyone is expected to complete > > > their integration testing by RC date. In fact I had sent numerous > > reminders > > > prior to the first RC starting from 2 weeks before the proposed RC > date. > > > > > > That's not the point. The code is changing at a rapid pace. Mike, for > > > example, commented on tons of critical fixes going in right up until > > > the RC is cut. Then we cut some artifacts and give people 72 hours to > > > test and buy off. What I'm advocating is to lengthen the process, > > > and not tie it to a timeline until we have better testing that > > > stabilizes our master. At that time, when people can trust master > > > remotely, then maybe individuals will take the time to poke at it > > > prior to RC. Maybe that's a horrible idea, but let's at least talk > > > about doing something until we're stable... or do we think we can > > > accomplish that in a timely fashion? > > > > > > I think there are a few subgroups in our team here. 1) people whose > > > job it is to develop on cloudstack, but don't really use it, 2) people > > > who use cloudstack daily, and only do development to bugfix and/or add > > > a pet feature. There may be some overlap for some individuals. This > > > process might work great for individuals whose job it is to focus on > > > cloudstack every single day and are tightly integrated with the > > > massive changes, but the rest of us who consume cloudstack don't > > > always have time to look at the big picture and focus on the unstable > > > branches. We use the releases and focus on making the stable ones > > > better and/or fixing/adding our pet features, until the next stable > > > one comes around. Until the development branches stabilize I don't > > > believe it will work for the users, they won't get involved until the > > > end. > > > > > > For me, personally, it's a waste of time to even look at a branch that > > > probably won't work due to sweeping changes that tend to occur between > > > releases. Make your core changes, add spring, replace the storage > > > subsystem, whatever it is, and then I'll go back and see what it broke > > > after the bugs are worked out in all of that. That's how group #2 > > > thinks, in general. And right now the only indicator that we're to > > > that point is when we start talking RC, at which point I have a 3 day > > > window that I hopefully catch and have time to play with it. > > > > > > > > > > > > > > > > > My impression from your responses Animesh is that you feel everything > > > is fine as-is. I don't know how anyone could think that given what > > > we've seen over the last two releases, especially you who had to cut > > > six RCs. We're blowing past our "time based releases", and trying to > > > push through buggy releases (for some reason). My intent was to sum up > > > and focus on some of the comments I've seen over the past few weeks > > > about low/sporadic RC participation, major changes going on at the > > > last minute, etc. I guess I'm in the minority though, since we're the > > > only ones discussing it. > > > > > > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > > cloud<http://solidfire.com/solution/overview/?video=play> > > *™* > > > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
