+1, on the automation test. When will we have good automation test?
> -----Original Message----- > From: David Nalley [mailto:da...@gnsa.us] > Sent: Monday, April 22, 2013 7:52 PM > To: dev@cloudstack.apache.org > Cc: cloudstack-...@incubator.apache.org > Subject: Re: [DISCUSS] ACS Release 4 month v/s 6 month > > On Mon, Apr 22, 2013 at 5:19 PM, Animesh Chaturvedi > <animesh.chaturv...@citrix.com> wrote: > > Folks > > > > We started discussing 4 month v/s 6 month release cycle in a another > thread [1]. Since the subject of that thread was different, community may > not have participated in this important discussion fully. I am are bringing > this > discussion to its own thread. Here is the summary so far please refer to [1] > for more details. > > > > Summary of discussion: > > - Animesh pointed out the technical debt that we have accumulated so > > far needs extra time to resolve > > - David, Chip favor shorter release cycle of 4 month and keeping > > master always stable and in good quality and enhancing automation as a > > solution to reduce QA manual effort. A focused defect fixing activity > > may be needed to reduce technical debt > > - Will brought up several points in the discussion: He called out heavy > dependence on manual QA for a release and pointed out that manual QA > may not be always available to match up ACS release schedule. Release > overhead for 4 month release is still high and suggest that moving to 6 month > will save on release overhead and that time can be used for strengthening > automation. > > - Joe agrees partly in release overhead being significant for major > > release > > > > If I missed out any important point please feel free to bring into the > > thread. > > > > There were some other discussion in [1] on release planning conference > and chip's clarification on time based v/s feature based releases but we will > not discuss those in this thread. Community has agreed to time-based > release already. > > > > [1] http://markmail.org/thread/6suq2fhltdvgvcxd > > > > > Hi Animesh: > > I thought I would add a few comments. To the folks reading - I apologize for > the length. If you haven't started yet, you may want to get some coffee/tea. > You've been warned. :) > > I think there are two concerns people think about when talking about > changing the length of the release cycle. > > The first concern is workload. Generating a release has a certain fixed > amount of overhead. Writing release notes, running votes, and a key part of > the discussion currently is the QA cycle. Currently a large portion of our QA > cycle is manual testing, which means we need lots of fingers on keyboards to > get it done. > > The other concern is quality. We always want to try and put our best foot > forward, and minimize bugs, yet the very act of developing software, and > especially developing new features, introduces bugs. > > Before I start telling you my reasoning, I want to lay out something that only > hit me tonight. We've talked about how extending the release cycle only > extends the length of development. We've talked about keeping the length > of the development period (e.g. pre-code freeze) the same, and essentially > only extending the amount of time for QA. > That works for the first release after a move from 4 to 6 months. It falls > apart > there after though. The problem is that at code freeze we branch. Master > immediately becomes open for future feature development, and you've just > extended it by an additional 2 months, because your cycle is that much > longer. > > I personally don't think that either concern is addressed well by lengthening > the cycle. Let me explain why > > On the quality front - the immediate threat to quality is instability > inserted by > additional development/new features/improvements. By increasing the > length of time for that disruption to occur, we get more of it, as people will > continue adding things that might have been deferred to a later release. We > have code merged that, realistically, we won't know if it works properly until > QA has gone in and verified it, perhaps months later. The more paths > changed in a given release, the greater the propensity for failure. > > On the overhead front - the overhead of writing release notes and > generating releases goes down. In addition the 'basic' testing that QA does > now is additional overhead in each cycle as well. What isn't overhead though > is testing for new features. Because effectively longer cycles mean greater > amount of codebase change, more features, etc. The actual testing in a given > cycle must also increase. > > If we were really focused on absolute best quality we could deliver with our > current tooling, I believe the easiest would be to make our release cycle the > length of a single feature development period + the QA cycle (e.g. > shortening the release cycle.) > > Is there a right or wrong answer? There shouldn't be. Our software should be > deliverable at any point in the development cycle we choose. > 4 months or 6 months would not be a quality, or QA workload issue. We > would have automated test suites written for 80+% of our testing and we > would run comprehensive tests against propose merges to guarantee that an > incoming features both works as intended, and doesn't harm the rest of the > platform in the process. We'd make release cycle decisions based upon the > same kind of preferences that make us choose a monkey mascot over a polar > bear mascot. > > That sadly isn't where we are today, but I do think it illuminates a clear > path > of where we need to go. (more automated testing) > > So where do I fall on 4 months versus 6 months? > > I think we should stay the 4 month course. Heres why: > * We set the 4-month release cycle expectation already. In theory we've > been through 2 releases, but really only 4.1 - 4.0 was dominated by getting > into the incubator, getting releases okayed legally, etc. > It took us 7 months to release. I am loathe to change something we only > recently set, and have only been through once. I am also worried about the > signal that such rapid contrary decision changes would send to our users. > > * I am worried about the impact to quality of our releases. Greater > introduction of features means an increased threat to stability. It may not > impact us in the 4.2 cycle, but without massive automated QA it will impact us > going forward in the 4.3 and later time frames. > > > If you've made it thus far, I hope your tea/coffee was enjoyable. > Thanks for reading. :) > > --David
