>From our other thread I had a thought about our "only on request" tests.
Today, in theory: - The lightweight tests run automatically based on path matching. This is an approximate implementation of the ideal of running based on whether they could impact a test signal. - Heavyweight (and more flaky) tests run on request. A while ago, our "lightweight" tests were a huge monolith and very flaky (Python still is in this state I think). While I was splitting up monolithic "lightweight" Java SDK tests to make them run only on relevant paths, some of the "heavyweight" tests became small enough that they run automatically, so we also have: - Smaller integration tests (like DirectRunner doing SplunkIO) run automatically based on path matching. Danny mentioned the idea of changing the above to: - Heavyweight tests run only if the lightweight tests are healthy. Here's an idea I had about a combination of these that I wanted to know if anyone had seen it or thought of how it could happen or why it is a bad idea: - Heavyweight tests are *required but not automatically run* based on path matching. A status would show up indicating that the PR is not green until you request and pass the heavyweight tests. - When the PR is actually ready you request them. Then I would move even the small integration tests into that latter category. Incidentally this also could easily apply to non-hermetic tests that make our security posture more difficult, requiring a committer to approve running them. Is this possible? Good? Bad? Kenn
