Do we have anyone here who works at GitHub that could potentially help? An internal contact would be the easiest (e.g., I was that initial contact point to get us the CloudBees CI version of Jenkins here to ease our own Jenkins usage problems). Otherwise, I'd imagine that INFRA are likely to be important allies here in requesting GitHub do anything.
On Sun, 10 Jan 2021 at 11:08, Jarek Potiuk <ja...@potiuk.com> wrote: > > On Sun, Jan 10, 2021 at 5:28 PM Matt Sicker <boa...@gmail.com> wrote: > > > If we can get GA to handle our use case properly, that would be awesome. > > Being in the security engineering domain, though, I’m generally pessimistic > > about security, so please excuse my cynicism. > > > > I totally understand. i am a bit more optimistic, especially that we > potentially could > throw some heavyweight - like a number of Serious Apache project making a > common and coordinated action of "We can either praise GitHub for their > cooperation" or "They are not secure and not willing to improve". > That is a publicity they would either love (the former) or hate (the > latter). > > I am happy to help in any way I can - represent INFRA in talks, describe the > problems and propose solutions, word the "carrot" and "stick opttions and > even prepare how they could look like - I coudl take part in discussions > with GitHub, maybe even escalate this to Microsoft if they will not show > they are cooperating - butI have no legitimation for doing so. > I have no power to throw the weight of ASF in the discussion. But I would > love to do that and lead that if only I had this kind of power at least > delegated > to me and provided with the means of contacting GitHub and representing > the ASF (but I doubt anyone would give me that power, it is a bit risky as > with big power I would have no big responsibility. > > Tough call - I am not sure how else I can help INFRA/ASF to help me and > others. > > J. > > > > > > On Sun, Jan 10, 2021 at 03:43 Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > I have a feeling (though I cannot know for sure) > > > that you are underestimating the power of an organization like ASF in > > > actually 'stating' their requirements and 'expectations' towards GitHub. > > > > > > I am now an engineer, but I used to be CTO, CEO, Head of IT, Head of > > > Technology > > > and I know that a lot can be achieved by proper communication, stating > > your > > > expectations clearly and follow-up and pushing when you are dealing with > > > partners like that - and engineering excellence or security perfection is > > > not the only > > > the thing that matters. Usability, maintenance, streamlining development > > > matter and if you > > > have "good enough security", they are more important for users. > > > > > > I know if you look at it from an "infrastructure security Jenkins" point > > of > > > view - the Jenkins > > > you manage is superior when it comes to security. > > > This is perfectly clear, and I have no intention to question that or > > > disagree with you. > > > And yes - in this aspect I fully agree with you. > > > > > > But there are other aspects which I see (and try to explain). > > > While I deeply care about security (as probably you could see from my > > > earlier > > > communication). Just limiting the discussion to "who is more secure" is a > > > terrible, > > > terrible oversimplification. > > > > > > I encourage you to exercise empathy and see it from the side I was > > > explaining - > > > maintenance, features, integration, streamlining development. Those are > > > important > > > things for developers. Less important for security engineers of course, > > but > > > if > > > we can satisfy security, those are the things that matter. > > > > > > I think currently we have mitigations for all the security problems we > > > found at the project > > > level. Also (as I mentioned before) we will have good leverage - via > > social > > > media pressure > > > to push GA into solving those that are 'systemic' problems we found. They > > > are not > > > necessary for our project to solve, but it would simplify your life as > > you > > > take care of so > > > many projects. So the security bounties that I opened are not for me - > > they > > > are for the > > > ASF as a whole and for the security team of ASF. I exercised a lot of > > > empathy to your > > > team that rather than only solving my problem, I also spend time and > > effort > > > to push > > > GA into solving it for all ASF projects and in the way that ASF infra > > > security will be satisfied. > > > I did not have to do that. Yet I try to think about your needs there. > > > > > > And to be honest I expect something in return. Empathy and understanding > > > other needs > > > I have - performance, usability, streamlining development, minimum > > > engineering effort > > > to solve our problems is the least I can ask for. Help in dealing > > > with GitHub and > > > exercising ASF powers would be great. > > > > > > Maybe with GitHub, the problem is that organizations like ASF do not > > > exercise > > > their leverage and do not clearly state what is essential for them while > > > working with > > > partners like them? > > > > > > Did the ASF explicitly contacted GA and firmly stated that solving the > > > problem of > > > self-hosted runnines is an absolute top priority to solve our performance > > > issues? > > > > > > I do not know. > > > > > > Did anyone from ASF contacted GA and firmly stated that the two bounties > > I > > > created are essential for the security team to be able to provide > > security > > > for > > > the organization? > > > > > > I do not know. > > > > > > Did the ASF push GA in any way in this direction stating that > > > ASF is considering alternatives? (The "stick" in this discussion) > > > > > > I do not know. > > > > > > Did the ASF propose GA that we can endorse their service, write blogs, > > and > > > ask > > > the 100s of projects that will use GA to endorse their service publicly > > > once they > > > start addressing our firmly stated needs and expectations? (The "carrot" > > in > > > this > > > discussion) > > > > > > I do not know. > > > > > > This is what I would do if I were at INFRA. I am not. I am not even an > > ASF > > > member to > > > have more insight and visibility into it. > > > > > > The only thing I can do is to ask for help and see if the ASF Infra is > > > willing to help in > > > the situation by exercising the powers that I do not have. > > > > > > For me, this is really a test, whether the ASF has the power to negotiate > > > with such > > > partners. If not - maybe it's time to think that everything (including > > > GitHub repos) > > > should be self-hosted by INFRA, because if you are dealing with partners > > > like that > > > you should have some negotiating power, otherwise, you put yourself in a > > > loosing > > > position. > > > > > > But again - I do not know much. This is what I would do If I had the > > > powers. > > > On my side, I think I've shown that I do above and beyond what you might > > > expect > > > from a PMC of one of the ASF projects, and asking for help from the > > > organization, > > > I - so far - proudly belong to, is the only thing left I have. I run out > > > of all ammo. > > > > > > So again - please help! > > > > > > J. > > > > > > > > > On Sat, Jan 9, 2021 at 11:00 PM Matt Sicker <boa...@gmail.com> wrote: > > > > > > > I work on the Jenkins security team. We don’t have embarrassing > > security > > > > failures like this anymore, but part of that is due to the added > > > complexity > > > > of a secure configuration. By the time GA meets your security > > standards, > > > > it’ll likely either require non-trivial changes to your CI scripts, or > > > > it’ll break various use cases that you otherwise considered to be > > > usability > > > > enhancements. It’s really getting annoying how every complaint you have > > > > about every non-Jenkins system isn’t a problem in Jenkins. We have more > > > > expertise available to customize things in such a way that works for > > > > non-proprietary SaaS that most services are optimized for (which is why > > > > their security models tend to fall short once a large organization like > > > > Apache tries using something). > > > > > > > > Many of the features you’re asking from GA are likely non-trivial > > > > architecture changes they’ll have to make to accommodate the > > non-trivial > > > > use cases we have. Or maybe it isn’t and they’re just incompetent? > > > > > > > > On Sat, Jan 9, 2021 at 05:58 Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > The multiple threads about how shitty those are in practice for > > > your > > > > > > > needs seem to indicate otherwise. Security and easy learning > > curves > > > > > > > don't seem to get along too well, do they? > > > > > > > > > > > > > > > > The usabilty, integration level (especially GitHub Actions), > > > maintenance > > > > > effort needed > > > > > - thi is far, far superior. If only we could solve one simple > > problem - > > > > > securely running > > > > > the self-hosted runners for GA - all our problems are solved > > > INSTANTLY. > > > > > > > > > > Security issues happen everywhere, at least if they happen in such > > > > services > > > > > you can > > > > > mitigate (we just did it in Airflow- we mitigated all the security > > > issues > > > > > we found), > > > > > open bounty requests (I did - I opened two bounty requests) and then > > > > > escalate. > > > > > If I do not hear about my 2 security bounties from GitHub shortly, > > > > > I am going to start a hell of a social media campaign about it > > > > > using all the means I can. I tried to responsibly disclose it but I > > am > > > > > going to write a nice > > > > > blog post about "How to exploit Github Actions" and I am going to > > tell > > > > them > > > > > that before > > > > > I publish it and give them a chance to fix it. > > > > > > > > > > So you have many ways to influence the security of public services > > like > > > > > that. I think it's > > > > > much better than when you have to manage security yourselves. > > > > > > > > > > > > > > > > > > > > > > > > That would all be possible in Jenkins, some of it would be fairly > > > > > > > simple to integrate, others would indeed be non-trivial rewrites. > > > > > > > > > > > > > > > > > > > > > > > Yep. The non-trivial ones I am afraid of. It took me a year to > > perfect > > > > and > > > > > optimise > > > > > a number of steps in our CI and the problem was - it worked really > > well > > > > > until it stopped > > > > > because of uncontrolled increase of usage from other projects and no > > > > secure > > > > > way > > > > > to add extra resources needed (even if we have all the funds - we now > > > > have > > > > > 8000 USD > > > > > secured from our stakeholders - with outlook for more) to run those. > > > But > > > > if > > > > > you add the > > > > > engineering effort needed to migrate, the engineering time for that > > > costs > > > > > FAR more than > > > > > just that - enabling compute resources to use all the engineering > > > efforts > > > > > you've already > > > > > spent. This is no brainer which way is simpler, cheaper and can be > > done > > > > > faster. > > > > > We just need to have a secure way of doing it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You can have your own Jenkins controller for your PMC. This is > > > vastly > > > > > > > simpler for you to administer than a super time-shared > > environment > > > > > > > like GA. CI systems seem to be a dime a dozen nowadays, yet not a > > > > > > > single one seems to implement job scheduling in a sufficiently > > > > > > > customizable way that scales. > > > > > > > > > > > > > > > > I do not want nor need to administer my CI. And I've done that many > > > times > > > > > in > > > > > the past - Jenkins, Bamboo, GitLab - you name it. > > > > > Heck - I built and maintained my first custom CI framework for my > > > company > > > > > some 20 years ago when the "CI" was just being coined. > > > > > With CI as a service, I do not want to do it anymore. At all. CI for > > me > > > > > should just 'be there'. > > > > > Great CI is one that you are not aware of its existence until your > > test > > > > > fail - and > > > > > even then you just want to see the logs of your failed tests and > > figure > > > > > out the reason > > > > > This is what you want from CI system. I do not want to learn how > > > > > to manage Jenkins, install plugins, configure that etc. This is not > > my > > > > job, > > > > > nor any > > > > > one in our project. This requires far more than just setting it > > > > > up - it is making sure that it is secure, that it runs 24/7, that it > > > gets > > > > > updated etc. etc. > > > > > This is far more complex than 'just use CI'. We have enough trouble > > > with > > > > > setting up > > > > > and maintaining runners (once we get them securely connected). > > > > > > > > > > I know it looks differently from the infrastructure person point of > > > view > > > > - > > > > > running > > > > > Jenkins is pretty much core part of what you do. But for project > > > > > developers, CI > > > > > should just 'work'. This is what I get from GitHub Actions. It just > > > > > 'works'. I have > > > > > to spend 0 effort to maintain it. Sometimes when it does not work, it > > > > > pains, but > > > > > then it's their problem to fix - and they have to fix it eventually > > > > because > > > > > they get > > > > > pressure from all their customers. In case I run my own jenkins > > install > > > > and > > > > > administer it - all those problems fall on us. I do not want that. > > This > > > > > moves us > > > > > away from doing what we should - develop our product. > > > > > > > > > > > > > > > > > > > > > > > > Definitely valid points. Any CI migrations are non-trivial, > > > > especially > > > > > > > once you've set up nice workflows. Perhaps there are some > > > > alternatives > > > > > > > that can help bridge the gap if GA still can't meet your needs. > > > I've > > > > > > > seen prow [1] used in various projects in the Kubernetes > > > communities, > > > > > > > and I'm sure there must be plenty of others. > > > > > > > > > > > > > > > > GA meets all my needs. Except one that I am asking ASF to help with - > > > > > make GitHub focus on making a secure way of working with self-hosted > > > > > runners. That's it . We even (In November) opened a PR to Github > > > Actions > > > > > Runner to enable it: https://github.com/actions/runner/pull/783 > > > > > But we have not heard anything since. > > > > > > > > > > This is what I ask INFRA to help with - put pressure on GitHub to > > make > > > it > > > > > happen. I need nothing more - no money, no Jenkins, nothing like > > that. > > > > > I just want to be able to spend the money we managed to secure. > > > > > > > > > > J. > > > > > > > > > > > > > > > -- > > > > > +48 660 796 129 > > > > > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > +48 660 796 129
