[
https://issues.apache.org/jira/browse/CASSANDRA-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ling Mao updated CASSANDRA-21008:
---------------------------------
Fix Version/s: 5.0.x
> Optimizing Randomized Testing for Reproducibility
> -------------------------------------------------
>
> Key: CASSANDRA-21008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21008
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Test/unit
> Reporter: Ling Mao
> Assignee: Ling Mao
> Priority: Normal
> Fix For: 5.0.x
>
>
> h4. Goals and Motivation
> Random Test is designed to help with testing systems that rely on randomness,
> such as those involving probabilistic behavior, concurrency, or randomized
> algorithms. It enables automated, repeatable testing of code that behaves
> differently on each run, ensuring more comprehensive test coverage.
> Key benefits:
> Test Uncertainty: It provides tools for testing systems where inputs,
> behaviors, or results vary across test runs.
> Reproducibility: It helps ensure tests can be reliably reproduced, even with
> randomness involved.
> Stress Testing: It is ideal for identifying edge cases and bugs that might
> not be easily found in deterministic tests.
> Better Test Coverage: Randomized tests provide more varied input
> combinations, leading to more thorough testing.
> In short, Random Test ensures the robustness and reliability of systems
> dealing with randomness, making it essential for testing complex or
> non-deterministic behavior.
> h4. Approach
> {code:java}
> private static final String RANDOMIZED_CONTEXT_FQN =
> "com.carrotsearch.randomizedtesting.RandomizedContext";
> Class.forName(RANDOMIZED_CONTEXT_FQN); // sets isRandomTestMode = true{code}
> We use this approach to determine whether we are in Random Test Mode or
> Normal Mode (if we can find the FQN, we activate the test infrastructure).
> {code:java}
> Randomness.java
> public static Random get() {
> if (isRandomTestMode()) {
> return our ThreadLocal seedable random implementation;
> } else {
> return java native Random implementation;
> }
> } {code}
> We centralize all random operations in *_Randomness.java_* to make them
> repeatable, unified, and aggregated (when in Random Test Mode, all randomness
> becomes reproducible and deterministic). For example:
> 1. Java Random APIs (SecureRandom, Collections.shuffle, UUID.randomUUID(),
> Math.random(), etc.)
> 2. Additional random utilities (randomUTF8, randomLocale, randomDate,
> randomAsciiLetters, etc.)
> We print the seed when a unit test fails:
> {code:java}
> java.lang.RuntimeException: java.lang.ExceptionInInitializerError
> at
> __randomizedtesting.SeedInfo.seed([FFB4961AF9935703:3AFD18F7B24FCA34]:0)
> at
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$7(Instance.java:675)
> ... {code}
> Then use the following command to reproduce the failure:
> {code:java}
> ant testsome -Dtest.name=org.apache.cassandra.db.marshal.MyRandomizedTest
> -Dtests.seed=FFB4961AF9935703 {code}
> We can also print JVM system properties, memory, and CPU information for
> additional debugging context.
> h4. Proposed Changes
> {code:java}
> // Before
> double randDbl = random.nextDouble();
> // After
> double randDbl = Randomness.get().nextDouble(); {code}
> h4. Test Plan
> We will refactor all random test suites. Existing test suites will verify
> these changes. Additionally, we will add unit tests for Randomness.
> h4. Rejected Alternatives: Why not use carrotsearch randomizedtesting
> directly?
> Elasticsearch and Lucene have successfully used randomizedtesting (Apache-2.0
> license) for decades([7]), and it's already present in Cassandra. However, we
> reject this approach for the following reasons:
> {*}1. Initialization timing issues{*}: Using Java reflection to call
> *_{{RandomizedContext.current().getRandom()}}_* causes
> {{java.lang.ExceptionInInitializerError}} due to class initialization timing
> at startup.
> {*}2. Boundary limitations{*}: randomizedtesting accesses random contexts
> within {{@BeforeClass}} and {{@AfterClass}} boundaries, but static test class
> initializers cannot access random contexts. However, we need random behavior
> during server startup.
> {*}3. Distributed JVM incompatibility{*}: randomizedtesting is not compatible
> with distributed JVM tests. Our native Java approach works seamlessly with
> distributed JVM tests, simulators, and QuickTheories.
> *3.1 Possible* {*}approach{*}{*}:{*}
> For unit tests:
> {code:java}
> public class UUIDTypeTest extends RandomTestSuite
> @Listeners({ ReproduceInfoPrinter.class })
> @ThreadLeakScope(ThreadLeakScope.Scope.NONE)
> @SeedDecorators({ MixWithSuiteName.class })
> @TestMethodProviders(value = { JUnit4MethodProvider.class })
> public abstract class RandomTestSuite extends RandomizedTest{code}
> For distributed JVM tests:
> {code:java}
> @RunWith(RandomizedRunner.class)
> @Listeners({ ReproduceInfoPrinter.class })
> @ThreadLeakScope(ThreadLeakScope.Scope.NONE)
> @SeedDecorators({ MixWithSuiteName.class })
> public class HintDataReappearingTest extends AbstractHintWindowTest{code}
> {*}3.1.1 Limitations of this p{*}{*}ossible{*} *approaches:*
> Some distributed JVM tests cannot obtain {{RandomizedContext}}
> Tests using {{@RunWith(Parameterized.class)}} or
> {{@RunWith(BMUnitRunner.class)}} cannot work with
> {{@RunWith(RandomizedRunner.class)}} since only one {{@RunWith}} annotation
> is allowed per test
> Extending {{RandomTestSuite}} or using {{@RunWith(RandomizedRunner.class)}}
> requires a base test class, causing extensive code changes that are difficult
> to review
> h4. Reference
> [1] [https://labs.carrotsearch.com/randomizedtesting-concept.html]
> [2]
> [https://get.carrotsearch.com/presentations/dweiss-barcelona-testing-2011.pdf]
> [3]
> [https://spinscale.de/posts/2020-04-22-testing-and-releasing-elasticsearch-and-the-elastic-stack.html]
> [4] [https://github.com/randomizedtesting/randomizedtesting]
> [5] [https://labs.carrotsearch.com/randomizedtesting.html]
> [6]
> [https://github.com/randomizedtesting/randomizedtesting/?tab=Apache-2.0-1-ov-file#]
> [7]https://github.com/elastic/elasticsearch/blob/004fab6627b0ae65cf939e0d69a4db8dbd175798/server/src/main/java/org/elasticsearch/common/Randomness.java#L93
> [8] [https://www.youtube.com/watch?v=zD57QKzqdCw]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
