I have a feeling (though I cannot know for sure) that you are underestimating the power of an organization like ASF in actually 'stating' their requirements and 'expectations' towards GitHub.
I am now an engineer, but I used to be CTO, CEO, Head of IT, Head of Technology and I know that a lot can be achieved by proper communication, stating your expectations clearly and follow-up and pushing when you are dealing with partners like that - and engineering excellence or security perfection is not the only the thing that matters. Usability, maintenance, streamlining development matter and if you have "good enough security", they are more important for users. I know if you look at it from an "infrastructure security Jenkins" point of view - the Jenkins you manage is superior when it comes to security. This is perfectly clear, and I have no intention to question that or disagree with you. And yes - in this aspect I fully agree with you. But there are other aspects which I see (and try to explain). While I deeply care about security (as probably you could see from my earlier communication). Just limiting the discussion to "who is more secure" is a terrible, terrible oversimplification. I encourage you to exercise empathy and see it from the side I was explaining - maintenance, features, integration, streamlining development. Those are important things for developers. Less important for security engineers of course, but if we can satisfy security, those are the things that matter. I think currently we have mitigations for all the security problems we found at the project level. Also (as I mentioned before) we will have good leverage - via social media pressure to push GA into solving those that are 'systemic' problems we found. They are not necessary for our project to solve, but it would simplify your life as you take care of so many projects. So the security bounties that I opened are not for me - they are for the ASF as a whole and for the security team of ASF. I exercised a lot of empathy to your team that rather than only solving my problem, I also spend time and effort to push GA into solving it for all ASF projects and in the way that ASF infra security will be satisfied. I did not have to do that. Yet I try to think about your needs there. And to be honest I expect something in return. Empathy and understanding other needs I have - performance, usability, streamlining development, minimum engineering effort to solve our problems is the least I can ask for. Help in dealing with GitHub and exercising ASF powers would be great. Maybe with GitHub, the problem is that organizations like ASF do not exercise their leverage and do not clearly state what is essential for them while working with partners like them? Did the ASF explicitly contacted GA and firmly stated that solving the problem of self-hosted runnines is an absolute top priority to solve our performance issues? I do not know. Did anyone from ASF contacted GA and firmly stated that the two bounties I created are essential for the security team to be able to provide security for the organization? I do not know. Did the ASF push GA in any way in this direction stating that ASF is considering alternatives? (The "stick" in this discussion) I do not know. Did the ASF propose GA that we can endorse their service, write blogs, and ask the 100s of projects that will use GA to endorse their service publicly once they start addressing our firmly stated needs and expectations? (The "carrot" in this discussion) I do not know. This is what I would do if I were at INFRA. I am not. I am not even an ASF member to have more insight and visibility into it. The only thing I can do is to ask for help and see if the ASF Infra is willing to help in the situation by exercising the powers that I do not have. For me, this is really a test, whether the ASF has the power to negotiate with such partners. If not - maybe it's time to think that everything (including GitHub repos) should be self-hosted by INFRA, because if you are dealing with partners like that you should have some negotiating power, otherwise, you put yourself in a loosing position. But again - I do not know much. This is what I would do If I had the powers. On my side, I think I've shown that I do above and beyond what you might expect from a PMC of one of the ASF projects, and asking for help from the organization, I - so far - proudly belong to, is the only thing left I have. I run out of all ammo. So again - please help! J. On Sat, Jan 9, 2021 at 11:00 PM Matt Sicker <boa...@gmail.com> wrote: > I work on the Jenkins security team. We don’t have embarrassing security > failures like this anymore, but part of that is due to the added complexity > of a secure configuration. By the time GA meets your security standards, > it’ll likely either require non-trivial changes to your CI scripts, or > it’ll break various use cases that you otherwise considered to be usability > enhancements. It’s really getting annoying how every complaint you have > about every non-Jenkins system isn’t a problem in Jenkins. We have more > expertise available to customize things in such a way that works for > non-proprietary SaaS that most services are optimized for (which is why > their security models tend to fall short once a large organization like > Apache tries using something). > > Many of the features you’re asking from GA are likely non-trivial > architecture changes they’ll have to make to accommodate the non-trivial > use cases we have. Or maybe it isn’t and they’re just incompetent? > > On Sat, Jan 9, 2021 at 05:58 Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > > > > > > The multiple threads about how shitty those are in practice for your > > > > needs seem to indicate otherwise. Security and easy learning curves > > > > don't seem to get along too well, do they? > > > > > > > The usabilty, integration level (especially GitHub Actions), maintenance > > effort needed > > - thi is far, far superior. If only we could solve one simple problem - > > securely running > > the self-hosted runners for GA - all our problems are solved INSTANTLY. > > > > Security issues happen everywhere, at least if they happen in such > services > > you can > > mitigate (we just did it in Airflow- we mitigated all the security issues > > we found), > > open bounty requests (I did - I opened two bounty requests) and then > > escalate. > > If I do not hear about my 2 security bounties from GitHub shortly, > > I am going to start a hell of a social media campaign about it > > using all the means I can. I tried to responsibly disclose it but I am > > going to write a nice > > blog post about "How to exploit Github Actions" and I am going to tell > them > > that before > > I publish it and give them a chance to fix it. > > > > So you have many ways to influence the security of public services like > > that. I think it's > > much better than when you have to manage security yourselves. > > > > > > > > > > > > That would all be possible in Jenkins, some of it would be fairly > > > > simple to integrate, others would indeed be non-trivial rewrites. > > > > > > > > > > > Yep. The non-trivial ones I am afraid of. It took me a year to perfect > and > > optimise > > a number of steps in our CI and the problem was - it worked really well > > until it stopped > > because of uncontrolled increase of usage from other projects and no > secure > > way > > to add extra resources needed (even if we have all the funds - we now > have > > 8000 USD > > secured from our stakeholders - with outlook for more) to run those. But > if > > you add the > > engineering effort needed to migrate, the engineering time for that costs > > FAR more than > > just that - enabling compute resources to use all the engineering efforts > > you've already > > spent. This is no brainer which way is simpler, cheaper and can be done > > faster. > > We just need to have a secure way of doing it. > > > > > > > > > > > > > > > > You can have your own Jenkins controller for your PMC. This is vastly > > > > simpler for you to administer than a super time-shared environment > > > > like GA. CI systems seem to be a dime a dozen nowadays, yet not a > > > > single one seems to implement job scheduling in a sufficiently > > > > customizable way that scales. > > > > > > > I do not want nor need to administer my CI. And I've done that many times > > in > > the past - Jenkins, Bamboo, GitLab - you name it. > > Heck - I built and maintained my first custom CI framework for my company > > some 20 years ago when the "CI" was just being coined. > > With CI as a service, I do not want to do it anymore. At all. CI for me > > should just 'be there'. > > Great CI is one that you are not aware of its existence until your test > > fail - and > > even then you just want to see the logs of your failed tests and figure > > out the reason > > This is what you want from CI system. I do not want to learn how > > to manage Jenkins, install plugins, configure that etc. This is not my > job, > > nor any > > one in our project. This requires far more than just setting it > > up - it is making sure that it is secure, that it runs 24/7, that it gets > > updated etc. etc. > > This is far more complex than 'just use CI'. We have enough trouble with > > setting up > > and maintaining runners (once we get them securely connected). > > > > I know it looks differently from the infrastructure person point of view > - > > running > > Jenkins is pretty much core part of what you do. But for project > > developers, CI > > should just 'work'. This is what I get from GitHub Actions. It just > > 'works'. I have > > to spend 0 effort to maintain it. Sometimes when it does not work, it > > pains, but > > then it's their problem to fix - and they have to fix it eventually > because > > they get > > pressure from all their customers. In case I run my own jenkins install > and > > administer it - all those problems fall on us. I do not want that. This > > moves us > > away from doing what we should - develop our product. > > > > > > > > > > > > Definitely valid points. Any CI migrations are non-trivial, > especially > > > > once you've set up nice workflows. Perhaps there are some > alternatives > > > > that can help bridge the gap if GA still can't meet your needs. I've > > > > seen prow [1] used in various projects in the Kubernetes communities, > > > > and I'm sure there must be plenty of others. > > > > > > > GA meets all my needs. Except one that I am asking ASF to help with - > > make GitHub focus on making a secure way of working with self-hosted > > runners. That's it . We even (In November) opened a PR to Github Actions > > Runner to enable it: https://github.com/actions/runner/pull/783 > > But we have not heard anything since. > > > > This is what I ask INFRA to help with - put pressure on GitHub to make it > > happen. I need nothing more - no money, no Jenkins, nothing like that. > > I just want to be able to spend the money we managed to secure. > > > > J. > > > > > > -- > > +48 660 796 129 > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
