> > On Tue, 23 Jan 2007 17:54:10 -0500, Diego Novillo <[EMAIL PROTECTED]> said: > > > So, I was doing some archeology on past releases and we seem to be > > getting into longer release cycles. > > Interesting. > > I'm a GCC observer, not a participant, but here are some thoughts: > > As far as I can tell, it looks to me like there's a vicious cycle > going on. Picking an arbitrary starting point: >
Let me bring up another point: 0) bugs go unnoticed for a couple of releases and then become part of the release criteria. This is 0) what really causes stage 3 to become long rather than what you mention in 1). Most of the bugs introduced during stage 1 that are caught during stage 3 or earlier are fixed quicker than the ones which were caught after the .2 release was done. > Now, the good news is that this cycle can be a virtuous cycle rather > than a vicious cycle: if you can lower one of these measurements > (length of stage 3, size of branches, size of patches, number of > bugs), then the other measurements will start going down. "All" you > have to do is find a way to mute one of the links somehow, focus on > the measurement at the end of that link, and then things will start > getting better. I don't think this will work because more bugs will just be found, the longer stage 3 is. > It's not obvious to what the best way is to do that, but here are some > ideas. Taking the links one by one: > > 1: Either fix bugs faster, or release with more bugs. > > 2: Artificially shorten the lifespan of development branches somehow, > so that big branches don't appear during stage 3. We already did this for 4.2 which is one reason why 4.2 has almost no new features and why a lot of patches were left out. > 3: Throttle the size of patches: don't let people do gigantic > merges, no matter the size of the branch. This is wrong as the gigantic merges are needed in some cases to be able change the infrastuture of GCC. A good example is recently (and soon) are mem-ssa, GIMPLE_MODIFY_STMT, and dataflow. All really could not be done by simple little patches. Tree-ssa was another example. > 4: Don't have buggy code in your branches: improve code quality of > development branches somehow. Some of the bigger branches actually have requirements for merging. Both the tree-ssa and dataflow branch have a couple of requirements, this will not find all bugs/issues. Remember GCC runs on around 30 processors (not counting variants on each one or even OSs). Running on all targets would take a month plus fixes and then making sure you don't break all of them. > For link 3, you'd change the rules to alternate between stage 1 and > stage 3 on a fast basis (no stage 2 would be necessary): do a small > merge (of a portion of a branch, if necessary), shake out bugs, and > repeat. Concretely, you could have two rules in GCC's development > process: Even though I proposed this before, I don't think it will help unless people are testing GCC with lots of code daily or right after a big merge. > * Patches more than a certain size aren't allowed. This won't work, see above. > * No patches are allowed if there are more than X release-blocking > bugs outstanding. (For some small value of X; 0 is one > possibility.) I don't think this will work out because you are punishing all developers while one developer gets his/her act together. In some cases, they could have just had some bad news about their mother. > With this, the trunk is almost always in a releasable state; you can > release almost whenever you want to, since you'd basically be at the > end of stage 3 every week, or every day, or every hour. Moving to > these rules would be painful, but once you start making progress, I > bet you'd find that, for example, the pressures leading to long-lived > branches will diminish. (Not go away, but diminish.) I don't think so because most of the regressions which are being reported are actually after even the .1 release. > For 4, you should probably spend some time figuring out why bugs are > being introduced into the code in the first place. Is test coverage > not good enough? If so, why - do people not write enough tests, is it > hard to write good enough tests, something else? Is the review > process inadequate? If so, why: are rules insufficiently stringent, > are reviewers sloppy, are there not enough reviewers, are patches too > hard to review? Some cases is because test coverage is not good enough in general, C++. Other cases, you just did not think about a corner case. Even in other cases, you exposed a latent bug in another part of the code which did not think about a corner case. > My guess is that most or all of those are factors, but some are more > important than others. My favorite tactic to decrease the number of > bugs is to set up a unit test framework for your code base (so you can > test changes to individual functions without having to run the whole > compiler), and to strongly encourage patches to be accompanied by unit > tests. This does not catch corner cases which is where most of the issues. This gets us back to the original problem, where you can think of inputs that should be valid but no, there are other inputs which are valid. Unit testing I think can help out in some cases like reload (there was an unit test posted before, I don't have the URL right now). But that was not really an unit test, it was more like testing a specific target with reload changes with out testing the rest of the compiler. That still did not catch the corner cases, it did catch a lot of bugs but only because that one target (sh) relied on reload a lot. I don't have a solution for either the long release cycle anymore as I am starting to see releases are a huge thing. That take a long time and either we need to stick to the current idea of 4.2 or scrap it. One idea to lower down the release cycle time is have specific QA people to test a whole distro at the end of stage1 and then again at the end of stage2. This will get the time down for reporting of major bugs. We will still end up with the problem that all code is ran through so we get bug reports after the .1 release that now block the next .0 release from happening. Sorry about the length of this email. Thanks, Andrew Pinski
