> 250 iterations isn't enough; I use 500 as a low water mark. Say more here. I originally had it at 500 but neither Mick nor I knew why and figured we could suss this out on this thread.
> This is also assuming that circle and ASF CI run the same tests, which > is not entirely true. +1: we need to fix this. My intuition is the path to getting circle-ci in parity on coverage is a shorter path than getting ASF CI to 3 green runs for GA. That consistent w/your perception as well or do you disagree? On Mon, Sep 26, 2022, at 2:15 PM, Brandon Williams wrote: > On Mon, Sep 26, 2022 at 12:20 PM Josh McKenzie <jmcken...@apache.org> wrote: > > For CI on a patch, run the pre-commit suite and also run multiplexer with > > 250 runs on new, changed, or related tests to ensure not flaky > > 250 iterations isn't enough; I use 500 as a low water mark. > > > I think it's worth calling out: relying on circleci test runs along with > > ASF CI as an option implies we have the same confidence in a green circleci > > run as we have in a green ASF CI run. > > This is also assuming that circle and ASF CI run the same tests, which > is not entirely true. We know that circle isn't running some upgrade > tests from https://issues.apache.org/jira/browse/CASSANDRA-17912 but I > think we need to specifically investigate parity between the two > systems for further discrepancies before we can begin to compare > confidence in runs between them. >
