Yeah, agreed. Sijie
On Oct 18, 2017 2:38 PM, "Ivan Kelly" <iv...@apache.org> wrote: > We should also do that. But that alone doesn't give us an idea of > which tests are flaking the most. > > On Wed, Oct 18, 2017 at 2:30 PM, Sijie Guo <guosi...@gmail.com> wrote: > > some of the tests were already marked @FlakyTest. > > https://github.com/apache/bookkeeper/issues?q=flaky+is%3Aopen > > > > another idea is to have a issue label `FlakyTest`, create an issue when > you > > see a flake and label it as `FlakyTest`. > > > > - Sijie > > > > On Wed, Oct 18, 2017 at 1:26 PM, Ivan Kelly <iv...@apache.org> wrote: > > > >> Hi folks, > >> > >> We need to get a handle on the flakes we're in CI. I've yet to see a > >> build come back green on all runs. The flakes seem to be different > >> each time, so it's hard to see which to tackle first. > >> > >> To this end, I've started a spreadsheet with a league table for flakes. > >> https://docs.google.com/spreadsheets/d/1CaYA7b4YlUGev8JqXBP8-Q3L_ > >> HAmfwY46FgzIb7Qpss/edit#gid=0 > >> > >> When you see a flake, and you're sure it's a flake (i.e. you've seen > >> the test pass on the same commit locally), either add a line, or > >> increment the number if it's already there. > >> > >> This way we can identify which flakes are hurting us the most, and put > >> effort into dealing with them first. > >> > >> Cheers, > >> Ivan > >> >
