Tests fail due to variety of reasons. Some of them fail due to underlying infrastructural issues. For example, getting a clean run of Python DTests typically involves rerunning them a couple times. Is it possible to do that at the test framework level i.e. in Jenkins and/or CircleCI?
Dinesh > On Jan 24, 2020, at 12:52 PM, Jordan West <jorda...@gmail.com> wrote: > > That’s awesome that we have that set up. I was checking out b.a.o after my > email and noticed some recent runs. I don’t mean to prescribe any specific > way of surfacing results as long as they are easily accessible to all > contributors (well documented where to find them, etc). > > Progress on posting results to jira is also awesome. > > Thanks Mick! > > > Jordan > > > On Fri, Jan 24, 2020 at 12:24 PM Mick Semb Wever <m...@apache.org> wrote: > >>> In my opinion/experience, this is all a direct consequence of lack of >> trust in CI caused by flakiness. >> >> >> The challenge of this project's test state certainly feel like an >> insurmountable challenge at times… >> >> Having been battling away with Jenkins, because I do have ASF access and >> don't have premium CircleCI access, I've developed a bit of a routine for >> evaluating the Jenkins CI results the best I can for even the most trivial >> of patches, so I've got some input to this… >> >> A canonical record of test results is important, and we didn't have that >> until yesterday: take a look in bui...@cassandra.apache.org. It is now >> possible to search for commit SHAs and find their test results. >> >> And with the new pipeline builds these test results are summarised for all >> the different test build types. These summarised results also go to slack's >> #cassandra-builds channel. The summarised results contains a lot and I >> haven't completely verified them, any help would be appreciated there. The >> idea is also to also post these results back to the jira ticket. How to do >> that is already figured out. This was discussed in the 'Cassandra CI >> Status' thread and in CASSANDRA-15496. >> >> In addition, build failures (and the resuming success) for the 'artifacts' >> build step goes to the builds ML, and to the author (if their email address >> can be determined). As we stabilise the pipeline's builds, eg starting >> with unit tests, we could then more easily move into the "no broken >> windows" mode. >> >> >>> I also don't think it leads to the right behaviour or incentives. >> >> >> I agree that a gatekeeping approach won't work, we need instead to >> incentivise more reviewing, code cleaners, test fixers, documenters, etc. >> These actions should be praised and valued as much as any other. That said, >> a little blame often goes a long way. >> >> With all this^ said I don't see the need for special daily build with >> results posted to the dev ML. >> >> >> regards, >> Mick >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
