I've enabled the flag now Search for Flake on [1] to see what a flake output looks like. I'll write a script to collate these, so we can see what is flaking.
> Strictly tests (especially CI tests) must not be flaky. It really doesn't > carry much use. As everyone tends to ignore them. > Step-1: Separate them into a different pool, and you won't get a red mark > for failing this pool/group. > Step-2: Fix each test to be consistent and move it out of the optional pool. > Step-3: If there are any tests that can't be consistent but makes sense > (???) to keep them part of CI, we can leave them in this flaky/optional > pool. I like this idea of a quarantine for flakes. Do you know how to do this with junit/maven? -Ivan [1] https://builds.apache.org/job/bookkeeper_precommit_pullrequest_java8/179/org.apache.bookkeeper$bookkeeper-server/consoleText
