On 2026-02-04 18:34, Sebastien Lorquet wrote:
I have decided to work on distributed CI because Alan clearly listed
this as a tool that will help the community.
It wont be fancy initially, but it can be improved later. Release
early, release often, yeah?
So I am writing a tool that will allow many clients to fetch and report
jobs, in a way that can run on ANY machine with python and build tools.
If I may, I'd like to suggest few things for consideration.
"For safety, only approved users can retrieve jobs." I think it's worth
to not tie asking people for help to some pre-selected trustworthy users
(and let's be honest, how many people in the community do you truly know
so you can safely consider them to be trustworthy. Or maybe slightly
different scenario - if someone trustworthy asks you for access, how
sure are you that the asking person is truly the person you know.)
What I am aiming at is that you could also consider model used by BOINC
framework - their tasks are distributed to multiple workers, the result
is compared and only considered valid if it matches. SHA256 checksum of
the resulting binary could be used for this validation. (Alebeit from
security standpoint, there is still a problem in that without enough
users participating, someone can simply create multiple accounts and
fish for getting the same job.)
You could also combine the two approaches and consider some users more
trustworthy than others. Probably want to have random usernames too. Or
better yet - have the users identify themselves by signing the request
with a key (of which the server will know the public part.) But in short
- make offering help as easy as possible (some people may be too shy to
request access.)
You may also want to have worker ID in the request to make the server be
able to recognize that two requests from single user came from two
workers (and not from one worker which crashed and lost the first job.)
Another thing that probably could prove useful in the request is
(optional?) list of targets which the worker can process. For example,
it's going to be pretty easy to make your machine capable of building
for x86 but maybe not so much for some ARM chip that needs external
libraries from the manufacturer. (For example, the machine may have a
policy that only software from distro repository may be installed on
it.)
Other than that - the protocol seems ok to me at the first glance.
