Yep. We still continuously optimize it and we are reaching out to get funding for self-hosted runners. And I think it would be great to see that happening. I am happy to help anyone who needs some help there - I've been already helping Apache Beam with their GitHub Actions settings.
On Mon, Oct 19, 2020 at 6:12 AM Greg Stein <gst...@gmail.com> wrote: > This is some great news, Jarek. > > Given that GitHub build minutes are shared, we need more of this kind of > work from our many communities. > > Thanks, > Greg > InfraAdmin, ASF > > > On Sun, Oct 18, 2020 at 2:32 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > Hello Allen, > > > > I'd really love to give a try to Yetus - how it can actually make our > > approach better. > > > > I just merged the change I planned (finally we got to that), that > > implements the final optimisation that you mentioned. In the case of a > > single .md file change we got the build time down to about 1 minute, most > > of it being GitHub Actions "workflow" overhead. > > > > We went-down with the incremental pre-commit tests to ~ 25s. > > > > Build here: https://github.com/potiuk/airflow/pull/128/checks. As you > can > > see here: > > > > > https://github.com/potiuk/airflow/pull/128/checks?check_run_id=1268353637#step:7:98 > > in > > this case we run only the relevant static checks: > > > > - "No-tabs checker" > > - "Add license for all md files" > > - "Add TOC for md files." > > - "Check for merge conflicts" > > - "Detect Private Key" > > - "Fix End of Files" > > - "Trim Trailing Whitespace" > > - "Check for language that we do not accept as community", > > > > All the other checks, image building, and all the extra checks are > skipped > > (automatically as pre-commit determined them irrelevant). > > > > All this, while we keep really comprehensive tests and optimisation of > > image building for all the "serious steps". I tried to explain the > > philosophy and some basic assumptions behind our CI in > > https://github.com/apache/airflow/blob/master/CI.rst#ci-environment - > and > > I'd love to try to see how this plays together with the Yetus tool. > > > > Would it be possible to work together with the Yetus team on trying to > see > > how it can help to further optimise and possibly simplify the setup we > > have? I'd love to get some cooperation on those. I am nearly done with > all > > optimisations I planned, And we are for years (long before my tenure) > among > > top-3 Apache projects when it comes to CI-time use, so that might be a > good > > one if we can pull together some improvements. > > > > > > J. > > > > > > > > On Wed, Oct 14, 2020 at 4:41 PM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > Exactly - > dialectic vs. dislectic for example. > > > > > > On Wed, Oct 14, 2020 at 4:40 PM Jarek Potiuk <jarek.pot...@polidea.com > > > > > wrote: > > > > > >> And really sorry about yatus vs. yetus - I am slightly dialectic and > > when > > >> things are not in the dictionary, I tend to do many mistakes. I hope > > it's > > >> not something that people can take as a sign of being "worse", but if > > you > > >> felt offended by that - apologies. > > >> > > >> > > >> > > >> On Wed, Oct 14, 2020 at 4:34 PM Jarek Potiuk < > jarek.pot...@polidea.com> > > >> wrote: > > >> > > >>> Hey Allen, > > >>> > > >>> I would be super happy if you could help us to do it properly at > > Airlfow > > >>> - would you like to work with us and get the yatus configuration that > > >>> would work for us ? I am super happy to try it? Maybe you could open > PR > > >>> with some basic yatus implementation to start with and we could work > > >>> together to get it simplified? I would love to learn how to do it. > > >>> > > >>> J > > >>> > > >>> > > >>> On Wed, Oct 14, 2020 at 3:37 PM Allen Wittenauer > > >>> <a...@effectivemachines.com.invalid> wrote: > > >>> > > >>>> > > >>>> > > >>>> > On Oct 13, 2020, at 11:04 PM, Jarek Potiuk < > > jarek.pot...@polidea.com> > > >>>> wrote: > > >>>> > This is a logic > > >>>> > that we have to implement regardless - whether we use yatus or > > >>>> pre-commit > > >>>> > (please correct me if I am wrong). > > >>>> > > >>>> I'm not sure about yatus, but for yetus, for the most part, > > >>>> yes, one would like to need to implement custom rules in the > > personality to > > >>>> exactly duplicate the overly complicated and over engineered airflow > > >>>> setup. The big difference is that one wouldn't be starting from > > scratch. > > >>>> The difference engine is already there. The file filter is already > > there. > > >>>> full build vs. PR handling is already there. etc etc etc > > >>>> > > >>>> > For all others, this is not a big issue because in total all other > > >>>> > pre-commits take 2-3 minutes at best. And if we find that we need > to > > >>>> > optimize it further we can simply disable the '--all-files' switch > > for > > >>>> > pre-commit and they will only run on the latest commit-changed > files > > >>>> > (pre-commit will only run the tests related to those changed > files). > > >>>> But > > >>>> > since they are pretty fast (except pylint/mypy/flake8) we think > > >>>> running > > >>>> > them all, for now, is not a problem. > > >>>> > > >>>> That's what everyone thinks until they start aggregating the > > >>>> time across all changes... > > >>>> > > >>>> > > >>> > > >>> -- > > >>> > > >>> Jarek Potiuk > > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>> > > >>> M: +48 660 796 129 <+48660796129> > > >>> [image: Polidea] <https://www.polidea.com/> > > >>> > > >>> > > >> > > >> -- > > >> > > >> Jarek Potiuk > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > >> M: +48 660 796 129 <+48660796129> > > >> [image: Polidea] <https://www.polidea.com/> > > >> > > >> > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
