I work on the Jenkins security team. We don’t have embarrassing security failures like this anymore, but part of that is due to the added complexity of a secure configuration. By the time GA meets your security standards, it’ll likely either require non-trivial changes to your CI scripts, or it’ll break various use cases that you otherwise considered to be usability enhancements. It’s really getting annoying how every complaint you have about every non-Jenkins system isn’t a problem in Jenkins. We have more expertise available to customize things in such a way that works for non-proprietary SaaS that most services are optimized for (which is why their security models tend to fall short once a large organization like Apache tries using something).
Many of the features you’re asking from GA are likely non-trivial architecture changes they’ll have to make to accommodate the non-trivial use cases we have. Or maybe it isn’t and they’re just incompetent? On Sat, Jan 9, 2021 at 05:58 Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > > The multiple threads about how shitty those are in practice for your > > > needs seem to indicate otherwise. Security and easy learning curves > > > don't seem to get along too well, do they? > > > > The usabilty, integration level (especially GitHub Actions), maintenance > effort needed > - thi is far, far superior. If only we could solve one simple problem - > securely running > the self-hosted runners for GA - all our problems are solved INSTANTLY. > > Security issues happen everywhere, at least if they happen in such services > you can > mitigate (we just did it in Airflow- we mitigated all the security issues > we found), > open bounty requests (I did - I opened two bounty requests) and then > escalate. > If I do not hear about my 2 security bounties from GitHub shortly, > I am going to start a hell of a social media campaign about it > using all the means I can. I tried to responsibly disclose it but I am > going to write a nice > blog post about "How to exploit Github Actions" and I am going to tell them > that before > I publish it and give them a chance to fix it. > > So you have many ways to influence the security of public services like > that. I think it's > much better than when you have to manage security yourselves. > > > > > > > > That would all be possible in Jenkins, some of it would be fairly > > > simple to integrate, others would indeed be non-trivial rewrites. > > > > > > > Yep. The non-trivial ones I am afraid of. It took me a year to perfect and > optimise > a number of steps in our CI and the problem was - it worked really well > until it stopped > because of uncontrolled increase of usage from other projects and no secure > way > to add extra resources needed (even if we have all the funds - we now have > 8000 USD > secured from our stakeholders - with outlook for more) to run those. But if > you add the > engineering effort needed to migrate, the engineering time for that costs > FAR more than > just that - enabling compute resources to use all the engineering efforts > you've already > spent. This is no brainer which way is simpler, cheaper and can be done > faster. > We just need to have a secure way of doing it. > > > > > > > > > > > You can have your own Jenkins controller for your PMC. This is vastly > > > simpler for you to administer than a super time-shared environment > > > like GA. CI systems seem to be a dime a dozen nowadays, yet not a > > > single one seems to implement job scheduling in a sufficiently > > > customizable way that scales. > > > > I do not want nor need to administer my CI. And I've done that many times > in > the past - Jenkins, Bamboo, GitLab - you name it. > Heck - I built and maintained my first custom CI framework for my company > some 20 years ago when the "CI" was just being coined. > With CI as a service, I do not want to do it anymore. At all. CI for me > should just 'be there'. > Great CI is one that you are not aware of its existence until your test > fail - and > even then you just want to see the logs of your failed tests and figure > out the reason > This is what you want from CI system. I do not want to learn how > to manage Jenkins, install plugins, configure that etc. This is not my job, > nor any > one in our project. This requires far more than just setting it > up - it is making sure that it is secure, that it runs 24/7, that it gets > updated etc. etc. > This is far more complex than 'just use CI'. We have enough trouble with > setting up > and maintaining runners (once we get them securely connected). > > I know it looks differently from the infrastructure person point of view - > running > Jenkins is pretty much core part of what you do. But for project > developers, CI > should just 'work'. This is what I get from GitHub Actions. It just > 'works'. I have > to spend 0 effort to maintain it. Sometimes when it does not work, it > pains, but > then it's their problem to fix - and they have to fix it eventually because > they get > pressure from all their customers. In case I run my own jenkins install and > administer it - all those problems fall on us. I do not want that. This > moves us > away from doing what we should - develop our product. > > > > > > > > Definitely valid points. Any CI migrations are non-trivial, especially > > > once you've set up nice workflows. Perhaps there are some alternatives > > > that can help bridge the gap if GA still can't meet your needs. I've > > > seen prow [1] used in various projects in the Kubernetes communities, > > > and I'm sure there must be plenty of others. > > > > GA meets all my needs. Except one that I am asking ASF to help with - > make GitHub focus on making a secure way of working with self-hosted > runners. That's it . We even (In November) opened a PR to Github Actions > Runner to enable it: https://github.com/actions/runner/pull/783 > But we have not heard anything since. > > This is what I ask INFRA to help with - put pressure on GitHub to make it > happen. I need nothing more - no money, no Jenkins, nothing like that. > I just want to be able to spend the money we managed to secure. > > J. > > > -- > +48 660 796 129 >
