Hello Allen, I'd really love to give a try to Yetus - how it can actually make our approach better.
I just merged the change I planned (finally we got to that), that implements the final optimisation that you mentioned. In the case of a single .md file change we got the build time down to about 1 minute, most of it being GitHub Actions "workflow" overhead. We went-down with the incremental pre-commit tests to ~ 25s. Build here: https://github.com/potiuk/airflow/pull/128/checks. As you can see here: https://github.com/potiuk/airflow/pull/128/checks?check_run_id=1268353637#step:7:98 in this case we run only the relevant static checks: - "No-tabs checker" - "Add license for all md files" - "Add TOC for md files." - "Check for merge conflicts" - "Detect Private Key" - "Fix End of Files" - "Trim Trailing Whitespace" - "Check for language that we do not accept as community", All the other checks, image building, and all the extra checks are skipped (automatically as pre-commit determined them irrelevant). All this, while we keep really comprehensive tests and optimisation of image building for all the "serious steps". I tried to explain the philosophy and some basic assumptions behind our CI in https://github.com/apache/airflow/blob/master/CI.rst#ci-environment - and I'd love to try to see how this plays together with the Yetus tool. Would it be possible to work together with the Yetus team on trying to see how it can help to further optimise and possibly simplify the setup we have? I'd love to get some cooperation on those. I am nearly done with all optimisations I planned, And we are for years (long before my tenure) among top-3 Apache projects when it comes to CI-time use, so that might be a good one if we can pull together some improvements. J. On Wed, Oct 14, 2020 at 4:41 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Exactly - > dialectic vs. dislectic for example. > > On Wed, Oct 14, 2020 at 4:40 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> And really sorry about yatus vs. yetus - I am slightly dialectic and when >> things are not in the dictionary, I tend to do many mistakes. I hope it's >> not something that people can take as a sign of being "worse", but if you >> felt offended by that - apologies. >> >> >> >> On Wed, Oct 14, 2020 at 4:34 PM Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >>> Hey Allen, >>> >>> I would be super happy if you could help us to do it properly at Airlfow >>> - would you like to work with us and get the yatus configuration that >>> would work for us ? I am super happy to try it? Maybe you could open PR >>> with some basic yatus implementation to start with and we could work >>> together to get it simplified? I would love to learn how to do it. >>> >>> J >>> >>> >>> On Wed, Oct 14, 2020 at 3:37 PM Allen Wittenauer >>> <a...@effectivemachines.com.invalid> wrote: >>> >>>> >>>> >>>> > On Oct 13, 2020, at 11:04 PM, Jarek Potiuk <jarek.pot...@polidea.com> >>>> wrote: >>>> > This is a logic >>>> > that we have to implement regardless - whether we use yatus or >>>> pre-commit >>>> > (please correct me if I am wrong). >>>> >>>> I'm not sure about yatus, but for yetus, for the most part, >>>> yes, one would like to need to implement custom rules in the personality to >>>> exactly duplicate the overly complicated and over engineered airflow >>>> setup. The big difference is that one wouldn't be starting from scratch. >>>> The difference engine is already there. The file filter is already there. >>>> full build vs. PR handling is already there. etc etc etc >>>> >>>> > For all others, this is not a big issue because in total all other >>>> > pre-commits take 2-3 minutes at best. And if we find that we need to >>>> > optimize it further we can simply disable the '--all-files' switch for >>>> > pre-commit and they will only run on the latest commit-changed files >>>> > (pre-commit will only run the tests related to those changed files). >>>> But >>>> > since they are pretty fast (except pylint/mypy/flake8) we think >>>> running >>>> > them all, for now, is not a problem. >>>> >>>> That's what everyone thinks until they start aggregating the >>>> time across all changes... >>>> >>>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
