Hi Jarek,
Thank you for the document. Could you tell us more about the "custom security layer" that you implemented? Regards Antoine. Le 08/02/2021 à 01:44, Jarek Potiuk a écrit : > For anyone following this thread - some update from the progress we have in > Airflow on building self-hosted infrastructure for GitHub actions. > > Ash from Airflow is really close to finalizing the work on a nice > auto-scaling framework for self-hosted workers, but also we checked what is > the best value for money we can get. > > I've run some analysis on the performance and tested my hypothesis (based > on earlier experiences) of significant optimisations we can get. > > I've finished my analysis of potential optimizations we can get on our CI > with the Self-Hosted runners that Ash created. I did some performance > testing and (very crude) comparison of "traditional approach" with Local > SSDs 2 CPU instances running the tests with something I already tested > several times on various CI arrangements - running tests with High-Memory > instances (8CPU 64 GB Mem) and running everything (including docker engine) > in "tmpfs" - huge ramdisk. > Seems that 1h 20 minutes of test running can be decreased 8x (!)using this > approach (and parallelising some tests) at the same time decreasing the > cost 2x (!). Yep. You heard right. We can have faster builds this way and > pay less for that. Seems that we will be able to decrease the time to run > all tests for one combination to 10 minutes from 1h20 minutes. > This is possible because Ash and his team did a great job on setting up > auto-scaling EC2 instance runners on our Amazon EC2 account (we have > credits from Amazon to run those jobs - also Astronomer offered donation to > keep it running ). Seems that by utilizing it we can not only pay less but > also get much faster builds. > > If you are interested - my document is here. Open for comments - happy to > add you as editors if you want (just send me your gmail address in priv). > It is rather crude, I had no time to put a bit more effort into it due to > some significant changes in my company, but it should be easy to compare > the values and see the actual improvements we can get. There are likely a > few shortcuts there and some of the numbers are "back-of-the-envelope" and > we are going to validate them even more when we implement all the > optimisations, but the conclusions should be pretty sound. > > https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit# > > J. > > > On Fri, Jan 8, 2021 at 10:02 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> >> We should be able to make an efficient query via GraphQL API right? I found >>> the REST API for actions to be a little underwhelming. >> >> >> That was the first thing I checked when we started looking at the stats. >> Unfortunately last time that I checked (and I even opened an issue for >> that to >> Github support) there was not a Github Actions GraphQL API. >> >> I got a GH support answer "Yeah we know GH API does not have >> GraphQL support yet, sorry". I think it has not changed since. >> >> >> We have tried to make our builds faster with more caching but it's not easy >>> since it's an embedded systems project we need to target a lot of >>> configurations and most changes impact all builds. >>> >> >> Indeed, I know how much of my time was spent on optimising Airflow GH >> usage. >> I think we eventually decreased the usage 10x or more. But it never >> helped, for a >> long as currently anyone even accidentally could block all the slots in >> almost no >> time at all. We have no organisation-wide way to block this and this is >> the problem. >> >> Right now I could: >> a) mine cryptocurrency using PRs to any Apache project >> b) block the queue for everone >> >> I do not have to be even an Apache committer to do that. It's enough if >> just open one PR >> which is well crafted and spins of 180 jobs that run for 6 hours. It's >> super-flawed. >> >> >>> >>> We too would like to would like to take advantage of our own runners but >>> more for the ability to do Hardware In the Loop testing but have avoided >>> it >>> for the reasons already mentioned. >>> >> >> Self-hosted runner for now seems to be the only "Reasonable" option but >> the security >> issues with the current runner are not allowing us to do it. >> >>> >>> --Brennan >>> >> >> >> -- >> +48 660 796 129 >> > >
