Hello, everyone. TL;DR: I'm looking for some help to get pull-request CI running distributed, i.e. on multiple hosts.
Quick history ------------- The repo-mirror-ci project pretty much started as a simple cronjob running on hardware contributed to the purpose by Todd Goodman (considering how much this hardware has done for Gentoo, I'd suggest giving him a honorary citizenship of Gentoo or something like that ;-)). I can't say they were terrible but they weren't perfect either. Due to lack of time, I mostly focused on improving them moderately and adding new features. Today, everything is still running on the same hardware. While the hardware is real good, and I'm doing some effort to improve the performance of the tools, I think it's quite close to its limits. Right now, it is running three tasks every 20 minutes: updating repo mirrors, running pkgcheck on Gentoo and processing one pull request. However, if those tasks delay beyond those 20 minutes, the poor man's scheduling based on cron + lockfiles may cause some of the executions to be skipped. In other words, the best case is processing 3 pull requests / h. This is starting to be no longer sufficient, and causing significant delays during periods of high contributor activity -- while my goal would be to provide the results ASAP -- 10-20 minutes would be perfect. The goal and needs ------------------ I think the best way forward would be to start distributing tasks between more systems. While I don't really have time to make whole repo-mirror-ci distributed, I think it should be feasible to start splitting pull request processing out of it. I would use some help to achieve that, esp. wrt distributed task scheduling. Can you think of any tools that could help me get the task done easily and with as little of reinventing the wheel as possible? Right now, I have just a lot of trivial shell script that checks pull request for changes, checks them out, runs 'pmaint regen', pkgcheck, publishes all kinds of results and statuses, and also compares QA results to determine new failures created by the PR [1]. As I see it, checking for changes and submitting the results should stay on the current host. However, it should be able to run all the actual work on slaves and retrieve the results from them. I would appreciate all the help with implementing that, plus possibly some degree of control to make it possible to update pkgcheck on slaves easily. Please let me know if you're interested in helping. Thanks. [1]:https://bitbucket.org/mgorny/repo-mirror-ci/src/master/pull-requests.job -- Best regards, Michał Górny
pgpU8Dli2RywM.pgp
Description: OpenPGP digital signature
