[
https://issues.apache.org/jira/browse/CASSANDRA-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
C. Scott Andreas updated CASSANDRA-21501:
-----------------------------------------
Description:
{{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} is
intermittently flaky in CI, which can be reproduced more reliably by
introducing resource contention.
{{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} runs an
infinite commit loop on node1's MISC stage, then calls close on the cluster.
{{cluster.close()}} waits 60s ({{{}waitOnFutures(1 MINUTE){}}}) for
{{{}node1.shutdown(){}}}; {{node1.shutdown()}} runs {{Stage.shutdownAndWait(1
MINUTE)}} = {{shutdownNow()}} (interrupts the MISC thread) +
{{{}awaitTermination(1 MINUTE){}}}.
The loop ignores the interrupt, so shutdown can only proceed once {{commit()}}
throws on its own.
Under CI load that exceeded 60s and produces an {{{}IllegalStateException:
Unterminated threads detected{}}}listing parked {{node1_isolatedExecutor}}
threads.
This can be reproduced on a development machine by introducing CPU saturation
and running the test:
{{# saturate cores, then run repeatedly}}
{{for i in $(seq 1 12); do yes > /dev/null & done}}
{{for i in $(seq 1 6); do}}
{{ ant test-jvm-dtest-some
-Dtest.name=org.apache.cassandra.distributed.test.tcm.CMSShutdownTest
-Dno-checkstyle=true}}
{{done}}
{{kill %1 2>/dev/null; pkill yes}}
{{}}
A local test that just exits the loop on Thread.isInterrupted() wasn't
sufficient to eliminate all flakiness in this test, but a second change that
enables only GOSSIP and NETWORK in the in-JVM dtest was.
Additional details and analysis (model-supported):
[https://github.com/cscotta/cassandra/pull/7]
was:
{{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} is
intermittently flaky in CI, which can be reproduced more reliably by
introducing resource contention.
{{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} runs an
infinite commit loop on node1's MISC stage, then calls close on the cluster.
{{cluster.close()}} waits 60s ({{{}waitOnFutures(1 MINUTE){}}}) for
{{{}node1.shutdown(){}}}; {{node1.shutdown()}} runs {{Stage.shutdownAndWait(1
MINUTE)}} = {{shutdownNow()}} (interrupts the MISC thread) +
{{{}awaitTermination(1 MINUTE){}}}.
The loop was {{{}for (;;) commit(...){}}}, which ignores the interrupt, so
shutdown could only proceed once {{commit()}} threw on its own.
Under CI load that exceeded 60s and produces an {{{}IllegalStateException:
Unterminated threads detected{}}}listing parked {{node1_isolatedExecutor}}
threads.
This can be reproduced on a development machine by introducing CPU saturation
and running the test:
{{# saturate cores, then run repeatedly}}
{{for i in $(seq 1 12); do yes > /dev/null & done}}
{{for i in $(seq 1 6); do}}
{{ ant test-jvm-dtest-some
-Dtest.name=org.apache.cassandra.distributed.test.tcm.CMSShutdownTest
-Dno-checkstyle=true}}
{{done}}
{{kill %1 2>/dev/null; pkill yes}}
{{}}
A local test that just exits the loop on Thread.isInterrupted() wasn't
sufficient to eliminate all flakiness in this test, but a second change that
enables only GOSSIP and NETWORK in the in-JVM dtest was.
Additional details and analysis (model-supported):
[https://github.com/cscotta/cassandra/pull/7]
> Flaky test: CMSShutdownTest#shutdownCMSCoincidingWithUnsuccessfulCommit
> throws under load
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21501
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21501
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Transactional Cluster Metadata
> Reporter: C. Scott Andreas
> Assignee: Sam Tunnicliffe
> Priority: Normal
>
> {{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} is
> intermittently flaky in CI, which can be reproduced more reliably by
> introducing resource contention.
> {{CMSShutdownTest.shutdownCMSCoincidingWithUnsuccessfulCommit}} runs an
> infinite commit loop on node1's MISC stage, then calls close on the cluster.
> {{cluster.close()}} waits 60s ({{{}waitOnFutures(1 MINUTE){}}}) for
> {{{}node1.shutdown(){}}}; {{node1.shutdown()}} runs {{Stage.shutdownAndWait(1
> MINUTE)}} = {{shutdownNow()}} (interrupts the MISC thread) +
> {{{}awaitTermination(1 MINUTE){}}}.
> The loop ignores the interrupt, so shutdown can only proceed once
> {{commit()}} throws on its own.
> Under CI load that exceeded 60s and produces an {{{}IllegalStateException:
> Unterminated threads detected{}}}listing parked {{node1_isolatedExecutor}}
> threads.
> This can be reproduced on a development machine by introducing CPU saturation
> and running the test:
> {{# saturate cores, then run repeatedly}}
> {{for i in $(seq 1 12); do yes > /dev/null & done}}
> {{for i in $(seq 1 6); do}}
> {{ ant test-jvm-dtest-some
> -Dtest.name=org.apache.cassandra.distributed.test.tcm.CMSShutdownTest
> -Dno-checkstyle=true}}
> {{done}}
> {{kill %1 2>/dev/null; pkill yes}}
> {{}}
> A local test that just exits the loop on Thread.isInterrupted() wasn't
> sufficient to eliminate all flakiness in this test, but a second change that
> enables only GOSSIP and NETWORK in the in-JVM dtest was.
> Additional details and analysis (model-supported):
> [https://github.com/cscotta/cassandra/pull/7]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
