I’m not sure we have lots of bandwidth for upgrading existing tests anyway. However, the source of flakiness in existing tests is primarily either environmental or poor test design (relying on timings being a major culprit). If Harry were to produce flakiness it would have a higher likelihood of being real problems, and they would be reproducible if the tests were deterministic.
The Simulator on the other hand might be helpful for flaky tests, by being deterministic. We might want to develop a JUnitRunner that is backed by the simulator so we can easily switch it on to help diagnose flaky tests, and also for improved testing of concurrent code unit tests. We probably would not want to use it for all tests, however, as it might well slow down execution. From: Stefan Miklosovic <stefan.mikloso...@instaclustr.com> Date: Friday, 18 February 2022 at 08:57 To: dev@cassandra.apache.org <dev@cassandra.apache.org> Subject: Re: Apache Cassandra fuzz testing Benjamin's email could be written by myself :) Fully agree. On Fri, 18 Feb 2022 at 09:42, Benjamin Lerer <b.le...@gmail.com> wrote: > > Thanks a lot for raising that topic Alex. > > I did not have the chance to use Harry yet and I guess it is the case for > most of us. > Starting to use it in our new tests makes total sense to me. > I am more concerned about starting to migrate/update existing tests. It took > us time to build some reliable and non flaky tests to guarantee the > correctness of the codebase. As far as I can see from Harry's documentation > some features are still missing. The people lack experience with this tool > and it will take a bit of time for them to build that knowledge. Along the > way we might also discover some issues with Harry that need to be addressed. > > So I am +1 for starting to use it in our new tests and build our knowledge of > Harry. Regarding a migration of existing tests to it, I would wait a bit > before choosing to go down that path. > > > > Le mer. 16 févr. 2022 à 16:30, bened...@apache.org <bened...@apache.org> a > écrit : >> >> +1 >> >> >> >> The Simulator is hopefully going to be another powerful tool for this kind >> of work, and we should be encouraging the use of both for large or complex >> pieces of work. >> >> >> >> >> >> From: Alex Petrov <al...@coffeenco.de> >> Date: Wednesday, 16 February 2022 at 11:56 >> To: dev@cassandra.apache.org <dev@cassandra.apache.org> >> Subject: Re: Apache Cassandra fuzz testing >> >> (apologies for sending an incomplete email) >> >> >> >> Hi everyone, >> >> >> >> As you may know, we’ve been actively working on fuzz testing Apache >> Cassandra for the past several years and made quite a large progress on that >> front. >> >> >> >> We’ve cut a 0.0.1 release of Harry [1], a fuzz testing tool for apache >> Cassandra and merged CASSANDRA-16262 [2]. >> >> >> >> I’d recommend us as a community to take the next logical step and demand >> fuzz / property-based tests for all marjor patches, and start >> migrating/updating existing tests to be property-based rather than using >> hardoced values. >> >> >> >> Harry can be used to generate data, and then check that a sequence of events >> corresponds to Cassandra resolution rules. We will continue expanding Harry >> coverage and writing new models and checkers, too. >> >> >> >> If you would like to learn more about Harry, you can refer to a recent blog >> post [3]. I will also be happy to answer any questions you may have about >> Harry and assist you in writing your tests, and helping to extend Harry in >> case there’s a feature you may need to accomplish it. >> >> >> >> Thank you, >> >> —Alex >> >> >> >> [1] [GitHub - apache/cassandra-harry: Apache Cassandra - >> Harry](https://github.com/apache/cassandra-harry) >> >> [2] [CASSANDRA-16262 4.0 Quality: Coordination & Replication Fuzz Testing - >> ASF JIRA](https://issues.apache.org/jira/browse/CASSANDRA-16262) >> >> [3] [Apache Cassandra | Apache Cassandra >> Documentation](https://cassandra.apache.org/_/blog/Harry-an-Open-Source-Fuzz-Testing-and-Verification-Tool-for-Apache-Cassandra.html)
