Thanks Josh for bringing this up. I agree with Brandon on both points he made. I have also other examples of testing some things only in Jenkins or only in CircleCI. If people are fine with that, I can open a ticket and I just talked n Slack to Derek who expressed interest to help with that activity. Thanks in advance Derek, this will be great support for the project.
On Mon, 26 Sep 2022 at 14:15, Brandon Williams <dri...@gmail.com> wrote: > On Mon, Sep 26, 2022 at 12:20 PM Josh McKenzie <jmcken...@apache.org> > wrote: > > For CI on a patch, run the pre-commit suite and also run multiplexer > with 250 runs on new, changed, or related tests to ensure not flaky > > 250 iterations isn't enough; I use 500 as a low water mark. > > > I think it's worth calling out: relying on circleci test runs along with > ASF CI as an option implies we have the same confidence in a green circleci > run as we have in a green ASF CI run. > > This is also assuming that circle and ASF CI run the same tests, which > is not entirely true. We know that circle isn't running some upgrade > tests from https://issues.apache.org/jira/browse/CASSANDRA-17912 but I > think we need to specifically investigate parity between the two > systems for further discrepancies before we can begin to compare > confidence in runs between them. >
