C. Scott Andreas created CASSANDRA-21501:
--------------------------------------------
Summary: Flaky test:
CMSShutdownTest#shutdownCMSCoincidingWithUnsuccessfulCommit throws under load
Key: CASSANDRA-21501
URL: https://issues.apache.org/jira/browse/CASSANDRA-21501
Project: Apache Cassandra
Issue Type: Bug
Components: Transactional Cluster Metadata
Reporter: C. Scott Andreas
{{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} is
intermittently flaky in CI, which can be reproduced more reliably by
introducing resource contention.
{{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} runs an
infinite commit loop on node1's MISC stage, then calls close on the cluster.
{{cluster.close()}} waits 60s ({{{}waitOnFutures(1 MINUTE){}}}) for
{{{}node1.shutdown(){}}}; {{node1.shutdown()}} runs {{Stage.shutdownAndWait(1
MINUTE)}} = {{shutdownNow()}} (interrupts the MISC thread) +
{{{}awaitTermination(1 MINUTE){}}}.
The loop was {{{}for (;;) commit(...){}}}, which ignores the interrupt, so
shutdown could only proceed once {{commit()}} threw on its own.
Under CI load that exceeded 60s and produces an {{{}IllegalStateException:
Unterminated threads detected{}}}listing parked {{node1_isolatedExecutor}}
threads.
This can be reproduced on a development machine by introducing CPU saturation
and running the test:
{{# saturate cores, then run repeatedly}}
{{for i in $(seq 1 12); do yes > /dev/null & done}}
{{for i in $(seq 1 6); do}}
{{ ant test-jvm-dtest-some
-Dtest.name=org.apache.cassandra.distributed.test.tcm.CMSShutdownTest
-Dno-checkstyle=true}}
{{done}}
{{kill %1 2>/dev/null; pkill yes}}
{{}}
A local test that just exits the loop on Thread.isInterrupted() wasn't
sufficient to eliminate all flakiness in this test, but a second change that
enables only GOSSIP and NETWORK in the in-JVM dtest was.
Additional details and analysis (model-supported):
[https://github.com/cscotta/cassandra/pull/7]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
