OK, just realizing a low-tech way this has been done in my experience: have a workflow that goes red unless the commit / PR message to say something like "TESTED=link to workflow run". I don't love it, and it is easy to get lazy and circumvent, but I can see why it is what people chose to do.
A variation on that could be an integration test workflow that runs unconditionally based on the path, but immediately fails early unless the PR contains some [recently added] magic phrase or comment like "Run large tests". When the author is ready, they can lick in and re-run failed jobs through the GHA UI. In theory the draft/ready status could be used for this but I doubt that practice would catch on. Kenn On Wed, Oct 11, 2023 at 12:07 PM Danny McCormick via dev < dev@beam.apache.org> wrote: > I actually don't think GitHub supports path filters for required checks, > so you can't say something like "check X is only required if path Y or Z > are modified", you can only say "check X is required". I'm not 100% sure > on this, but it matches my memory and I could neither find docs with that > feature or get it to function like that on a personal repo. There are ways > to work around this (e.g. always status "good" when not on the path), but > they're messy. You also need a way of kicking off the job (today that is > comment triggers which is probably fine if these are limited to a few > checks per PR). > > Outside of the feasibility question, I'm at least theoretically > interested. This could allow us to turn some of our postcommits into > precommits without burning too much CI compute. I'm also generally +1 on > requiring more checks to pass before merging, especially if we can do so > through tooling. > > On Wed, Oct 11, 2023 at 10:42 AM Kenneth Knowles <k...@apache.org> wrote: > >> From our other thread I had a thought about our "only on request" tests. >> >> Today, in theory: >> >> - The lightweight tests run automatically based on path matching. This >> is an approximate implementation of the ideal of running based on whether >> they could impact a test signal. >> - Heavyweight (and more flaky) tests run on request. >> >> A while ago, our "lightweight" tests were a huge monolith and very flaky >> (Python still is in this state I think). While I was splitting up >> monolithic "lightweight" Java SDK tests to make them run only on relevant >> paths, some of the "heavyweight" tests became small enough that they run >> automatically, so we also have: >> >> - Smaller integration tests (like DirectRunner doing SplunkIO) run >> automatically based on path matching. >> >> Danny mentioned the idea of changing the above to: >> >> - Heavyweight tests run only if the lightweight tests are healthy. >> >> Here's an idea I had about a combination of these that I wanted to know >> if anyone had seen it or thought of how it could happen or why it is a bad >> idea: >> >> - Heavyweight tests are *required but not automatically run* based on >> path matching. A status would show up indicating that the PR is not green >> until you request and pass the heavyweight tests. >> - When the PR is actually ready you request them. >> >> Then I would move even the small integration tests into that latter >> category. Incidentally this also could easily apply to non-hermetic tests >> that make our security posture more difficult, requiring a committer to >> approve running them. >> >> Is this possible? Good? Bad? >> >> Kenn >> >
