Nate, I leave it to you to forward what-you-chose to the board@'s thread.
> Are there still troubles and what are they? TL;DR the ASF could provide the Cassandra community with an isolated jenkins installation: so that we can manage and control the Jenkins master, as well as ensure all donated hardware for Jenkins agents are dedicated and isolated to us. The long writeup… For Cassandra's use of ASF's Jenkins I see the following problems. ** Lack of trust (aka reliability) The Jenkins agents re-use their workspaces, as opposed to using new containers per test run, leading to broken agents, disks, git clones, etc. One broken test run, or a broken agent, too easily affects subsequent test executions. The complexity (and flakiness) around our tests is a real problem. CI on a project like Cassandra is a beast and the community is very limited in what it can do, it really needs the help of larger companies. Effort is required in fixing the broken, the flakey, and the ignored tests. Parallelising the tests will help by better isolating failures, but tests (and their execution scripts) also need to be better at cleaning up after themselves, or a more container approach needs to be taken. Another issue is that other projects sometimes using the agents, and Infra sometimes edits our build configurations (out of necessity). ** Lack of resources (throughput and response) Having only 9 agents: none of which can run the large dtests; is a problem. All 9 are from Instaclustr, much kudos! Three companies recently have said they will donate resources, this is work in progress. We have four release branches where we would like to provide per-commit post-commit testing. Each complete test execution currently take 24hr+. Parallelising tests atm won't help much as the agents are generally saturated (with the pipelines doing the top-level parallelisation). Once we get more hardware in place: for the sake of improving throughput; it will make sense to look into parallelising the tests more. The throughput of tests will also improve with effort put into removing/rewriting long running and inefficient tests. Also, and i think this is LHF, throughput could be improved by using (or taking inspiration from) Apache Yetus so to only run tests on what it relevant in the patch/commit. Ref: http://yetus.apache.org/documentation/0.11.1/precommit-basic/ ** Difficulty in use Jenkins is clumsy to use compared to the CI systems we use more often today: Travis, CircleCI, GH Actions. One of the complaints has been that only committers can kick off CI for patches (ie pre-commit CI runs). But I don't believe this to be a crucial issue for a number of reasons. 1. Thorough CI testing of a patch only needs to happen during the review process, to which a committer needs to be involved in anyway. 2. We don't have enough jenkins agents to handle the amount of throughput that automated branch/patch/pull-request testing would require. 3. Our tests could allow unknown contributors to take ownership of the agent servers (eg via the execution of bash scripts). 4. We have CircleCI working that provides basic testing for work-in-progress patches. Focusing on post-commit CI and having canonical results for our release branches, i think then it boils down to the stability and throughput of tests, and the persistence and permanence of results. The persistence and permanence of results is a bug bear for me. It has been partially addressed with posting the build results to the builds@ ML. But this only provides a (pretty raw) summary of the results. I'm keen to take the next step of the posting of CI results back to committed jira tickets (but am waiting on seeing Jenkins run stable for a while). If we had our own Jenkins master we could then look into retaining more/all build results. Being able to see the longer term trends of test results and well as execution times I hope would add the incentive to get more folk involved. Looping back to the ASF and what they could do: it would help us a lot in improving the stability and usability issues by providing us an isolated jenkins. Having our own master would simplify the setup, use and debugging, of Jenkins. It would still require some sunk cost but hopefully we'd end up with something better tailored to our needs. And with isolated agents help restore confidence. regards, Mick PS i really want to hear from those that were involved in the past with cassci, your skills and experience on this topic surpass anything i got. On Sun, 2 Feb 2020, at 22:51, Nate McCall wrote: > Hi folks, > The board is looking for feedback on CI infrastructure. I'm happy to take > some (constructive) comments back. (Shuler, Mick and David Capwell > specifically as folks who've most recently wrestled with this a fair bit). > > Thanks, > -Nate > > ---------- Forwarded message --------- > From: Dave Fisher <w...@apache.org> > Date: Mon, Feb 3, 2020 at 8:58 AM > Subject: [CI] What are the troubles projects face with CI and Infra > To: Apache Board <bo...@apache.org> > > > Hi - > > It has come to the attention of the board through looking at past board > reports that some projects are having problems with CI infrastructure. > > Are there still troubles and what are they? > > Regards, > Dave > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
