iamsanjay commented on PR #2467: URL: https://github.com/apache/solr/pull/2467#issuecomment-2122297638
> Hmmm [TestInterleavingTransformer](https://ge.apache.org/scans/tests?search.relativeStartTime=P90D&search.rootProjectNames=solr-root&search.timeZoneId=America%2FNew_York&tests.container=org.apache.solr.ltr.response.transform.TestInterleavingTransformer) failed and it isn't a flapper: Yup it's new. But ran successfully on my machine, tried multiple times now. Should we consider enabling the re-running failing test strategy, or I do not know If it's enabled already or not, to check whether it's flaky or not? https://docs.gradle.com/develocity/flaky-test-detection/ Evidence To further support it: # [Flaky Tests at Google and How We Mitigate Them](https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html) > We even have a way to denote a test as flaky - causing it to report a failure only if it fails 3 times in a row. This reduces false positives, but encourages developers to ignore flakiness in their own tests unless their tests start failing 3 times in a row, which is hardly a perfect solution # [Cost of Flaky Tests in CI: An Industrial Case Study](https://conf.researchr.org/details/icst-2024/icst-2024-industry/1/Cost-of-Flaky-Tests-in-CI-An-Industrial-Case-Study) > Automatically rerunning a test costs 0.02~cents, while not rerunning and thus letting the pipeline fail results in a manual investigation costing $5.67 in our context. The insights gained from our case study have led to the decision to shift effort from investigation and repair to automatically rerunning tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
