On Sun, Jan 10, 2021 at 5:28 PM Matt Sicker <boa...@gmail.com> wrote:
> If we can get GA to handle our use case properly, that would be awesome. > Being in the security engineering domain, though, I’m generally pessimistic > about security, so please excuse my cynicism. > I totally understand. i am a bit more optimistic, especially that we potentially could throw some heavyweight - like a number of Serious Apache project making a common and coordinated action of "We can either praise GitHub for their cooperation" or "They are not secure and not willing to improve". That is a publicity they would either love (the former) or hate (the latter). I am happy to help in any way I can - represent INFRA in talks, describe the problems and propose solutions, word the "carrot" and "stick opttions and even prepare how they could look like - I coudl take part in discussions with GitHub, maybe even escalate this to Microsoft if they will not show they are cooperating - butI have no legitimation for doing so. I have no power to throw the weight of ASF in the discussion. But I would love to do that and lead that if only I had this kind of power at least delegated to me and provided with the means of contacting GitHub and representing the ASF (but I doubt anyone would give me that power, it is a bit risky as with big power I would have no big responsibility. Tough call - I am not sure how else I can help INFRA/ASF to help me and others. J. > > On Sun, Jan 10, 2021 at 03:43 Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > I have a feeling (though I cannot know for sure) > > that you are underestimating the power of an organization like ASF in > > actually 'stating' their requirements and 'expectations' towards GitHub. > > > > I am now an engineer, but I used to be CTO, CEO, Head of IT, Head of > > Technology > > and I know that a lot can be achieved by proper communication, stating > your > > expectations clearly and follow-up and pushing when you are dealing with > > partners like that - and engineering excellence or security perfection is > > not the only > > the thing that matters. Usability, maintenance, streamlining development > > matter and if you > > have "good enough security", they are more important for users. > > > > I know if you look at it from an "infrastructure security Jenkins" point > of > > view - the Jenkins > > you manage is superior when it comes to security. > > This is perfectly clear, and I have no intention to question that or > > disagree with you. > > And yes - in this aspect I fully agree with you. > > > > But there are other aspects which I see (and try to explain). > > While I deeply care about security (as probably you could see from my > > earlier > > communication). Just limiting the discussion to "who is more secure" is a > > terrible, > > terrible oversimplification. > > > > I encourage you to exercise empathy and see it from the side I was > > explaining - > > maintenance, features, integration, streamlining development. Those are > > important > > things for developers. Less important for security engineers of course, > but > > if > > we can satisfy security, those are the things that matter. > > > > I think currently we have mitigations for all the security problems we > > found at the project > > level. Also (as I mentioned before) we will have good leverage - via > social > > media pressure > > to push GA into solving those that are 'systemic' problems we found. They > > are not > > necessary for our project to solve, but it would simplify your life as > you > > take care of so > > many projects. So the security bounties that I opened are not for me - > they > > are for the > > ASF as a whole and for the security team of ASF. I exercised a lot of > > empathy to your > > team that rather than only solving my problem, I also spend time and > effort > > to push > > GA into solving it for all ASF projects and in the way that ASF infra > > security will be satisfied. > > I did not have to do that. Yet I try to think about your needs there. > > > > And to be honest I expect something in return. Empathy and understanding > > other needs > > I have - performance, usability, streamlining development, minimum > > engineering effort > > to solve our problems is the least I can ask for. Help in dealing > > with GitHub and > > exercising ASF powers would be great. > > > > Maybe with GitHub, the problem is that organizations like ASF do not > > exercise > > their leverage and do not clearly state what is essential for them while > > working with > > partners like them? > > > > Did the ASF explicitly contacted GA and firmly stated that solving the > > problem of > > self-hosted runnines is an absolute top priority to solve our performance > > issues? > > > > I do not know. > > > > Did anyone from ASF contacted GA and firmly stated that the two bounties > I > > created are essential for the security team to be able to provide > security > > for > > the organization? > > > > I do not know. > > > > Did the ASF push GA in any way in this direction stating that > > ASF is considering alternatives? (The "stick" in this discussion) > > > > I do not know. > > > > Did the ASF propose GA that we can endorse their service, write blogs, > and > > ask > > the 100s of projects that will use GA to endorse their service publicly > > once they > > start addressing our firmly stated needs and expectations? (The "carrot" > in > > this > > discussion) > > > > I do not know. > > > > This is what I would do if I were at INFRA. I am not. I am not even an > ASF > > member to > > have more insight and visibility into it. > > > > The only thing I can do is to ask for help and see if the ASF Infra is > > willing to help in > > the situation by exercising the powers that I do not have. > > > > For me, this is really a test, whether the ASF has the power to negotiate > > with such > > partners. If not - maybe it's time to think that everything (including > > GitHub repos) > > should be self-hosted by INFRA, because if you are dealing with partners > > like that > > you should have some negotiating power, otherwise, you put yourself in a > > loosing > > position. > > > > But again - I do not know much. This is what I would do If I had the > > powers. > > On my side, I think I've shown that I do above and beyond what you might > > expect > > from a PMC of one of the ASF projects, and asking for help from the > > organization, > > I - so far - proudly belong to, is the only thing left I have. I run out > > of all ammo. > > > > So again - please help! > > > > J. > > > > > > On Sat, Jan 9, 2021 at 11:00 PM Matt Sicker <boa...@gmail.com> wrote: > > > > > I work on the Jenkins security team. We don’t have embarrassing > security > > > failures like this anymore, but part of that is due to the added > > complexity > > > of a secure configuration. By the time GA meets your security > standards, > > > it’ll likely either require non-trivial changes to your CI scripts, or > > > it’ll break various use cases that you otherwise considered to be > > usability > > > enhancements. It’s really getting annoying how every complaint you have > > > about every non-Jenkins system isn’t a problem in Jenkins. We have more > > > expertise available to customize things in such a way that works for > > > non-proprietary SaaS that most services are optimized for (which is why > > > their security models tend to fall short once a large organization like > > > Apache tries using something). > > > > > > Many of the features you’re asking from GA are likely non-trivial > > > architecture changes they’ll have to make to accommodate the > non-trivial > > > use cases we have. Or maybe it isn’t and they’re just incompetent? > > > > > > On Sat, Jan 9, 2021 at 05:58 Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > > > > > > > > > > > > > > The multiple threads about how shitty those are in practice for > > your > > > > > > needs seem to indicate otherwise. Security and easy learning > curves > > > > > > don't seem to get along too well, do they? > > > > > > > > > > > > > The usabilty, integration level (especially GitHub Actions), > > maintenance > > > > effort needed > > > > - thi is far, far superior. If only we could solve one simple > problem - > > > > securely running > > > > the self-hosted runners for GA - all our problems are solved > > INSTANTLY. > > > > > > > > Security issues happen everywhere, at least if they happen in such > > > services > > > > you can > > > > mitigate (we just did it in Airflow- we mitigated all the security > > issues > > > > we found), > > > > open bounty requests (I did - I opened two bounty requests) and then > > > > escalate. > > > > If I do not hear about my 2 security bounties from GitHub shortly, > > > > I am going to start a hell of a social media campaign about it > > > > using all the means I can. I tried to responsibly disclose it but I > am > > > > going to write a nice > > > > blog post about "How to exploit Github Actions" and I am going to > tell > > > them > > > > that before > > > > I publish it and give them a chance to fix it. > > > > > > > > So you have many ways to influence the security of public services > like > > > > that. I think it's > > > > much better than when you have to manage security yourselves. > > > > > > > > > > > > > > > > > > > > That would all be possible in Jenkins, some of it would be fairly > > > > > > simple to integrate, others would indeed be non-trivial rewrites. > > > > > > > > > > > > > > > > > > > Yep. The non-trivial ones I am afraid of. It took me a year to > perfect > > > and > > > > optimise > > > > a number of steps in our CI and the problem was - it worked really > well > > > > until it stopped > > > > because of uncontrolled increase of usage from other projects and no > > > secure > > > > way > > > > to add extra resources needed (even if we have all the funds - we now > > > have > > > > 8000 USD > > > > secured from our stakeholders - with outlook for more) to run those. > > But > > > if > > > > you add the > > > > engineering effort needed to migrate, the engineering time for that > > costs > > > > FAR more than > > > > just that - enabling compute resources to use all the engineering > > efforts > > > > you've already > > > > spent. This is no brainer which way is simpler, cheaper and can be > done > > > > faster. > > > > We just need to have a secure way of doing it. > > > > > > > > > > > > > > > > > > > > > > > > > > You can have your own Jenkins controller for your PMC. This is > > vastly > > > > > > simpler for you to administer than a super time-shared > environment > > > > > > like GA. CI systems seem to be a dime a dozen nowadays, yet not a > > > > > > single one seems to implement job scheduling in a sufficiently > > > > > > customizable way that scales. > > > > > > > > > > > > > I do not want nor need to administer my CI. And I've done that many > > times > > > > in > > > > the past - Jenkins, Bamboo, GitLab - you name it. > > > > Heck - I built and maintained my first custom CI framework for my > > company > > > > some 20 years ago when the "CI" was just being coined. > > > > With CI as a service, I do not want to do it anymore. At all. CI for > me > > > > should just 'be there'. > > > > Great CI is one that you are not aware of its existence until your > test > > > > fail - and > > > > even then you just want to see the logs of your failed tests and > figure > > > > out the reason > > > > This is what you want from CI system. I do not want to learn how > > > > to manage Jenkins, install plugins, configure that etc. This is not > my > > > job, > > > > nor any > > > > one in our project. This requires far more than just setting it > > > > up - it is making sure that it is secure, that it runs 24/7, that it > > gets > > > > updated etc. etc. > > > > This is far more complex than 'just use CI'. We have enough trouble > > with > > > > setting up > > > > and maintaining runners (once we get them securely connected). > > > > > > > > I know it looks differently from the infrastructure person point of > > view > > > - > > > > running > > > > Jenkins is pretty much core part of what you do. But for project > > > > developers, CI > > > > should just 'work'. This is what I get from GitHub Actions. It just > > > > 'works'. I have > > > > to spend 0 effort to maintain it. Sometimes when it does not work, it > > > > pains, but > > > > then it's their problem to fix - and they have to fix it eventually > > > because > > > > they get > > > > pressure from all their customers. In case I run my own jenkins > install > > > and > > > > administer it - all those problems fall on us. I do not want that. > This > > > > moves us > > > > away from doing what we should - develop our product. > > > > > > > > > > > > > > > > > > > > Definitely valid points. Any CI migrations are non-trivial, > > > especially > > > > > > once you've set up nice workflows. Perhaps there are some > > > alternatives > > > > > > that can help bridge the gap if GA still can't meet your needs. > > I've > > > > > > seen prow [1] used in various projects in the Kubernetes > > communities, > > > > > > and I'm sure there must be plenty of others. > > > > > > > > > > > > > GA meets all my needs. Except one that I am asking ASF to help with - > > > > make GitHub focus on making a secure way of working with self-hosted > > > > runners. That's it . We even (In November) opened a PR to Github > > Actions > > > > Runner to enable it: https://github.com/actions/runner/pull/783 > > > > But we have not heard anything since. > > > > > > > > This is what I ask INFRA to help with - put pressure on GitHub to > make > > it > > > > happen. I need nothing more - no money, no Jenkins, nothing like > that. > > > > I just want to be able to spend the money we managed to secure. > > > > > > > > J. > > > > > > > > > > > > -- > > > > +48 660 796 129 > > > > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- +48 660 796 129
