Dear all, I spent some time these days looking into the Release Lifecycle document. As we keep on saying we approach Beta based on the Jira board, I was curious what is the exact borderline to cut it.
Looking at all the latest reports (thanks to everyone who was working on that; I think having an overview on what's going on is always a good thing), I have the feeling that the thing that prevents us primarily from cutting beta at the moment is flaky tests. According to the lifecycle document: - No flaky tests - All tests (Unit Tests and DTests) should pass consistently. A failing test, upon analyzing the root cause of failure, may be “ignored in exceptional cases”, if appropriate, for the release, after discussion in the dev mailing list." Now the related questions that popped up into my mind: - "ignored in exceptional cases" - examples? - No flaky tests according to Jenkins or CircleCI? Also, some people run the free tier, others take advantage of premium CircleCI. What should be the framework? - Furthermore, flaky tests with what frequency? (This is a tricky question, I know) In different conversations with colleagues from the C* community I got the impression that canonical suite (in this case Jenkins) might be the right direction to follow. To be clear, I am always checking any failures seen in any environment and I truly believe that they are worth it to be checked. Not advocating to skip anything! But also, sometimes I feel in many cases CircleCI could provide input worth tracking but less likely to be product flakes. Am I right? In addition, different people use different CircleCI config and see different output. Not to mention flaky tests on Mac running with two cores... Yes, this is sometimes the only way to reproduce some of the reported tests' issues... So my idea was to suggest to start tracking an exact Jenkins report maybe? Anything reported out of it also to be checked but potentially to be able to leave it for Beta in case we don't feel it shows a product defect. One more thing to consider is that the big Test epic is primarily happening in beta. Curious to hear what the community thinks about this topic. Probably people also have additional thoughts based on experience from the previous releases. How those things worked in the past? Any lessons learned? What is our "plan Beta"? Ekaterina Dimitrova e. ekaterina.dimitr...@datastax.com w. www.datastax.com
