Hello! We also have this wonderful tool: https://mtcga.gridgain.com/
It keeps track of tests which are currently flaky or under investigation, and given your change it will report non-flaky tests that failed, skipping all problematic tests. It talks to TeamCity so it can run tests for you, check their results and comment JIRA with "Bot Visa", see: https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite TC Bot" Since we don't have too many flaky tests thanks to Make TeamCity Green Again movement, it works reasonably well. A year ago we did not have any of this but now we do, and we were able to shave 80% of test-checking busy work. This is mostly thanks to Dmitry Pavlov. Regards, -- Ilya Kasnacheev ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski < stanislav_kozlov...@outlook.com>: > Hey there Ignite community, > > I contribute to a fellow open-source project - Apache Kafka - and there we > have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on > every Pull Request and due to test flakiness, almost all of them turn out > red with 1 or 2 tests (completely unrelated to the change in the PR) > failing. This has resulted in committers either ignoring them and merging > the changes or in the worst case rerunning the hour-long build until it > becomes green. > This test flakiness has also slowed down our releases significantly. > > In general, I was just curious to understand if this is a problem that > your project faces as well. Does your project have a lot of intermittently > failing tests, do you have any active process of addressing such tests > (during the initial review, after realizing it is flaky, etc). Any pointers > will be greatly appreciated! > > Thanks, > Stanislav > > > <http://aka.ms/weboutlook> >
