I will try to resopnd, but please keep in mind that all these terms are somewhat contextual.
I think long and burn tests are somewhat synonymous. But most long/burn tests that we have in-tree aren't actually that long. They are just long compared to the unit tests. I personally would call the test long when it runs for hours at least, but realistically for days. Fuzz tests are randomised tests that attempt to find issues in the system under test. Most of fuzz tests we wrote using Harry are also property-based: they are using a model checker to simulate an internal state of the system and check its responses with a simplified representation. Simulator tests are just tests that use our simulator framework, that executes a test against a cluster of nodes deterministically by fully serialising all of its operations. We also have a bunch of smaller simulations that simulate different scenarios: bounces, metadata changes, etc, without actually starting the cluster. Those are not simulator tests though. I have also used the word "simulate" in the context of model-checking, but also mostly to illustrate that it's all context-dependent. I personally believe that many tests, and test pipelines can (and probably should) be deprecated. But last time I brought this up, there was a bit of pushback, so I think before we can consider deprecation of tests that we think are redundant we will have to substantially improve adoption of the tools that allow better multiplexing. As regards configurations, I do not think it is necessary to re-run an entire set of u/d/injvm tests with vnode/trie/etc configurations, and instead these scenarios should be exercised by config permuation using a fuzzer. As experience (and several recent issues particularly) show - some important settings are never touched by any of the tests at all, and since tests are static, a chance of finding any issues with some combination of those is slim. Apart from what we already have (data and schema generators and failure injection), we now need configuration generators that will find interesting configurations and run randomly generated workflows against those, expecting any configuration of Cassandra to behave the same. I do find our test matrix a bit convoluted, and in my experience you spend way more time tweaking tests to work for all configurations after some code changes, and they find legimitate issues rather infrequently. We would probably be better off with a quick "sanity check" for major configurations per commit which, again, would exercise a common set of operations, combined with a comprehensive test suite which will try to cover as much ground as possible. Hope this helps. --Alex On Thu, Nov 30, 2023, at 10:25 AM, Jacek Lewandowski wrote: > Hi, > > I'm getting a bit lost - what are the exact differences between those test > scenarios? What are the criteria for qualifying a test to be part of a > certain scenario? > > I'm working a little bit with tests and build scripts and the number of > different configurations for which we have a separate target in the build > starts to be problematic, I cannot imagine how problematic it is for a new > contributor. > > It is not urgent, but we should at least have a plan on how to simplify and > unify things. > > I'm in favour of reducing the number of test targets to the minimum - for > different configurations I think we should provide a parameter pointing to > jvm options file and maybe to cassandra.yaml. I know that we currently do > some super hacky things with cassandra yaml for different configs - like > concatenting parts of it. I presume it is not necessary - we can have a > default test config yaml and a directory with overriding yamls; while > building we could have a tool which is able to load the default > configuration, apply the override and save the resulting yaml somewhere in > the build/test/configs for example. That would allows us to easily use those > yamls in IDE as well - currently it is impossible. > > What do you think? > > Thank you and my apologize for bothering about lower priority stuff while we > have a 5.0 release headache... > > Jacek >
