Hi, I have some experience with CI and am willing to help out, I'll talk to you on IRC.
Mathy On Dec 7, 2016 19:57, "Michał Górny" <mgo...@gentoo.org> wrote: > Hello, everyone. > > TL;DR: I'm looking for some help to get pull-request CI running > distributed, i.e. on multiple hosts. > > > Quick history > ------------- > > The repo-mirror-ci project pretty much started as a simple cronjob > running on hardware contributed to the purpose by Todd Goodman > (considering how much this hardware has done for Gentoo, I'd suggest > giving him a honorary citizenship of Gentoo or something like > that ;-)). I can't say they were terrible but they weren't perfect > either. Due to lack of time, I mostly focused on improving them > moderately and adding new features. > > Today, everything is still running on the same hardware. While > the hardware is real good, and I'm doing some effort to improve > the performance of the tools, I think it's quite close to its limits. > > Right now, it is running three tasks every 20 minutes: updating repo > mirrors, running pkgcheck on Gentoo and processing one pull request. > However, if those tasks delay beyond those 20 minutes, the poor man's > scheduling based on cron + lockfiles may cause some of the executions > to be skipped. > > In other words, the best case is processing 3 pull requests / h. This > is starting to be no longer sufficient, and causing significant delays > during periods of high contributor activity -- while my goal would be to > provide the results ASAP -- 10-20 minutes would be perfect. > > > The goal and needs > ------------------ > > I think the best way forward would be to start distributing tasks > between more systems. While I don't really have time to make whole > repo-mirror-ci distributed, I think it should be feasible to start > splitting pull request processing out of it. > > I would use some help to achieve that, esp. wrt distributed task > scheduling. > > Can you think of any tools that could help me get the task done easily > and with as little of reinventing the wheel as possible? Right now, I > have just a lot of trivial shell script that checks pull request for > changes, checks them out, runs 'pmaint regen', pkgcheck, publishes all > kinds of results and statuses, and also compares QA results to > determine new failures created by the PR [1]. > > As I see it, checking for changes and submitting the results should > stay on the current host. However, it should be able to run all > the actual work on slaves and retrieve the results from them. I would > appreciate all the help with implementing that, plus possibly some > degree of control to make it possible to update pkgcheck on slaves > easily. > > Please let me know if you're interested in helping. Thanks. > > [1]:https://bitbucket.org/mgorny/repo-mirror-ci/src/ > master/pull-requests.job > > -- > Best regards, > Michał Górny >
