I actually don't think GitHub supports path filters for required checks, so you can't say something like "check X is only required if path Y or Z are modified", you can only say "check X is required". I'm not 100% sure on this, but it matches my memory and I could neither find docs with that feature or get it to function like that on a personal repo. There are ways to work around this (e.g. always status "good" when not on the path), but they're messy. You also need a way of kicking off the job (today that is comment triggers which is probably fine if these are limited to a few checks per PR).
Outside of the feasibility question, I'm at least theoretically interested. This could allow us to turn some of our postcommits into precommits without burning too much CI compute. I'm also generally +1 on requiring more checks to pass before merging, especially if we can do so through tooling. On Wed, Oct 11, 2023 at 10:42 AM Kenneth Knowles <k...@apache.org> wrote: > From our other thread I had a thought about our "only on request" tests. > > Today, in theory: > > - The lightweight tests run automatically based on path matching. This is > an approximate implementation of the ideal of running based on whether they > could impact a test signal. > - Heavyweight (and more flaky) tests run on request. > > A while ago, our "lightweight" tests were a huge monolith and very flaky > (Python still is in this state I think). While I was splitting up > monolithic "lightweight" Java SDK tests to make them run only on relevant > paths, some of the "heavyweight" tests became small enough that they run > automatically, so we also have: > > - Smaller integration tests (like DirectRunner doing SplunkIO) run > automatically based on path matching. > > Danny mentioned the idea of changing the above to: > > - Heavyweight tests run only if the lightweight tests are healthy. > > Here's an idea I had about a combination of these that I wanted to know if > anyone had seen it or thought of how it could happen or why it is a bad > idea: > > - Heavyweight tests are *required but not automatically run* based on > path matching. A status would show up indicating that the PR is not green > until you request and pass the heavyweight tests. > - When the PR is actually ready you request them. > > Then I would move even the small integration tests into that latter > category. Incidentally this also could easily apply to non-hermetic tests > that make our security posture more difficult, requiring a committer to > approve running them. > > Is this possible? Good? Bad? > > Kenn >
