Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Jarek Potiuk
I think I understand well where yatus sits :). And pre-commit it is both - a bit earlier in the stack (i.e. where developer commits the code) but also works very well for the CI case. We use it with `--all-files` now though as this is maybe ~5% of all the time of our CI build time so we have not ye

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Allen Wittenauer
> On Oct 13, 2020, at 11:46 AM, Jarek Potiuk wrote: > > I am rather interested in how those kinds of cases might be handled better > by Yetus - i.e. how much smarter it can be when selecting which parts of > the tests should be run - and how you would define such relation. What > pre-commit is

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Jarek Potiuk
Pre-commit (when you run stuff on your local change) works only on the files that you changed (you can make it a bit more complex like " run always" etc" ). The pre-commit takes care of running those checks only on the files that are changed, it automatically splits the list of changed files into "

Re: GitHub PR comment build trigger

2020-10-13 Thread Matt Sicker
Could be that it requires a newer version of Jenkins like 2.249.x. Not sure on the exact requirements, but whenever something doesn't show updates in the UC, that's typically one of the main reasons. On Tue, 13 Oct 2020 at 11:09, Gavin McDonald wrote: > > Hi , > > On Tue, Oct 13, 2020 at 5:25 PM

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Allen Wittenauer
> On Oct 13, 2020, at 9:02 AM, Jarek Potiuk wrote: > > Yep having pre-commits is cool and we extensively use it as part of our > setup in Airflow. Since we are heavily Pythonic project we are using the > fantastic https://pre-commit.com/ framework. Is pre-commit still "dumb?" i.e.,

Re: GitHub PR comment build trigger

2020-10-13 Thread Gavin McDonald
Hi , On Tue, Oct 13, 2020 at 5:25 PM Matt Sicker wrote: > I'm not sure if we have the latest version of it, but the GitHub SCM > reporting plugin we have installed should add GitHub Checks links to > replay parts or all of a pipeline. Note that the GitHub SCM Reporting > option is not enabled by

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Jarek Potiuk
Yep having pre-commits is cool and we extensively use it as part of our setup in Airflow. Since we are heavily Pythonic project we are using the fantastic https://pre-commit.com/ framework. We have ~ 60 fast pre-commit checks that are run both as pre-commit hooks for developers and as CI verificat

Re: GitHub PR comment build trigger

2020-10-13 Thread Matt Sicker
I'm not sure if we have the latest version of it, but the GitHub SCM reporting plugin we have installed should add GitHub Checks links to replay parts or all of a pipeline. Note that the GitHub SCM Reporting option is not enabled by default, but enabling it is about all you need to do to get the fe

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Allen Wittenauer
> On Oct 13, 2020, at 5:00 AM, Jarek Potiuk wrote: > Who else is using GitHub Actions extensively? It's funny you mention that. A lot of the stress on the ASF Jenkins instance was helped tremendously by deploying Apache Yetus. It might be useful for some users on GitHub Actions to

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Jarek Potiuk
We are also exploring the possibility of having our own hosted runners. Regarding the management - I've learned recently that as of April you can manage secrets in the repository vi API - even if they are not present in the UI, without Infra doing it for you, so maybe self-hosted runners can also b

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Vladimir Sitnikov
>We have - apparently 180 concurrency limit for all Apache projects. Is there a way to get per project / per job breakdown? >including a custom cancel-workflow-run Action I developed: >https://github.com/potiuk/cancel-workflow-runs That sounds amazing. I often cancel runs manually, however, maki

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Antoine Pitrou
On Tue, 13 Oct 2020 14:00:17 +0200 Jarek Potiuk wrote: > > Who else is using GitHub Actions extensively? At Apache Arrow we are using GHA quite intensively as well. We have tens of different builds, which are triggered based on which files were modified. Some of them can be rather long.

GitHub Actions Concurrency Limits for Apache projects

2020-10-13 Thread Jarek Potiuk
Hello everyone, I've just opened https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20978 about it but I think it's best to start discussion here at builds as well. So I just copy what I wrote in the ticket: At Apache Airflow, we are starting to hit - more and more often - concurrency lim