We need our buildbots to work for branches. I propose the following changes, in decreasing order of priority.
1) If something like bindings is broken on a build bot for branches then disable the test on that buildbot. It is far better to disable bindings tests than it is to continue to allow the build bot to fail and thus encourage us to ignore the build bot entirely. 2) We should find a way to keep functioning build slaves for our branches that support the same tests as we run for trunk. If that means providing parallel installations of dependencies on the same machine or using separate machines then we should do that. 3) The build bot should not use the same slave for builds of branch and trunk changes. Right now the waterfall view does not give you a good view of the state of our branches/trunk. If I break 1.7.x the bot will show red until the next build which might be on trunk succeeds. That's not useful. I can understand sometimes running a test on a branch that's not normally tested (like I did with the 1.7.x-diff-translate branch) and using the 1.7.x slave. But we really ought to have slaves for each release branch we're using. Obviously 2 and 3 are longer term propositions. But #1 is something we should fix immediately. To that end I have conducted an audit of what's broken on which build bots and which release branches. My findings are below. 1.8.x: svn-x64-ubuntu-gcc: works svn-x64-centos-gcc: works bb-openbsd: doesn't build 1.8.x svn-windows-local: JavaHL fails. svn-windows-ra: Perl bindings fails. 1.7.x: svn-x64-ubuntu-gcc: Ruby bindings fail (build not just tests) svn-x64-centos-gcc: works bb-openbsd: doesn't build 1.8.x svn-windows-local: JavaHL fails. svn-windows-ra: Build fails (can't determine why bot is down right now). We should disable the swig-rb tests on 1.7.x on svn-x864-ubuntu-gcc since the problem there is that the version of Ruby on that host is too new for 1.7.x. Bert said the JavaHL tests on svn-windows-local are known due to some sort of DLL PATH problem. We should fix or disable these. I recall Bert having said in the past that SWIG-PL doesn't build on Windows properly, we should disable that test for swig-windows-ra. The build bots are down for Windows right now due to a network issue. So I can't run a specific test to determine what's wrong with svn-windows-ra and 1.7.x and the logs for the old builds have been removed already. We have two ways to handle turning tests off by branch. First we can pass the branch into the scripts that it's running for and then turn things off. The only problem with this is that svncheck.sh for the *nix buildbots already takes the list of tests to run. So we'll need to change the scripts to accept this in parallel with the master changes to pass it. We could alternatively detect the branch from the working copy. But I'm not sure how easy this will be to do. Or we could just skip all of the above and get the build bots setup with separate slaves for each branch and then they can have their own independent scripts. I'd start on some of the above steps but I'm also not clear on how these scripts get updated on the slaves. Does someone have to do that manually, are they automatically updated? If they are manually updated are the current versions in SVN? For that matter is there someplace where we document exactly who is responsible for which slaves?
