[
https://issues.apache.org/jira/browse/CASSANDRA-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281118#comment-17281118
]
Jacek Lewandowski commented on CASSANDRA-16415:
-----------------------------------------------
I encountered a weird problem - see my PR for 3.11:
https://github.com/apache/cassandra/pull/891
Basically there are no changes in the production code. I modified a little bit
the mixed read test mostly by adding gossip and networking features to the
cluster configuration (as gossiper is required for this test run). Apparently I
did something wrong because the cluster does not run properly and it complains
with the following exception (repeated many, many times):
{noformat}
[junit-timeout] ERROR 14:35:29 Unable to merge schema from /127.0.0.2
[junit-timeout] java.util.concurrent.RejectedExecutionException:
ThreadPoolExecutor has shut down
[junit-timeout] at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:58)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
[na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
[na:1.8.0_212]
[junit-timeout] at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:162)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:904)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:885)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:954)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:310)
~[dtest-3.11.11.jar:na]
[junit-timeout] at java.lang.Iterable.forEach(Iterable.java:75)
~[na:1.8.0_212]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:310)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1398)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1380)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.service.MigrationCoordinator.mergeSchemaFrom(MigrationCoordinator.java:367)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:404)
[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:393)
[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
[dtest-3.11.11.jar:na]
[junit-timeout] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_212]
[junit-timeout] at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
[dtest-3.11.11.jar:na]
[junit-timeout] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
[junit-timeout] ERROR [node1_InternalResponseStage:1] node1 2021-02-08
15:35:29,093 MigrationCoordinator.java:408 - Unable to merge schema from
/127.0.0.2
[junit-timeout] java.util.concurrent.RejectedExecutionException:
ThreadPoolExecutor has shut down
[junit-timeout] at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:58)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
[na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
[na:1.8.0_212]
[junit-timeout] at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:162)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:904)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:885)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:954)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:310)
~[dtest-3.11.11.jar:na]
[junit-timeout] at java.lang.Iterable.forEach(Iterable.java:75)
~[na:1.8.0_212]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:310)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1398)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1380)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.service.MigrationCoordinator.mergeSchemaFrom(MigrationCoordinator.java:367)
~[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:404)
[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:393)
[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
[dtest-3.11.11.jar:na]
[junit-timeout] at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
[dtest-3.11.11.jar:na]
[junit-timeout] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_212]
[junit-timeout] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_212]
[junit-timeout] at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
[dtest-3.11.11.jar:na]
[junit-timeout] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
{noformat}
Can someone help me a bit with this? (cc [~jwest], [~blerer])
> Broken test mixedModeReadColumnSubsetDigestCheck -
> org.apache.cassandra.distributed.upgrade.MixedModeReadTest
> -------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16415
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Test/dtest/java
> Reporter: Yifan Cai
> Assignee: Jacek Lewandowski
> Priority: Normal
> Fix For: 3.11.x, 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The test has been failing and can always be reproduced in the recent CI.
> Stack trace:
> {code:java}
> junit.framework.AssertionFailedError: Found Digest Mismatch
> at
> org.apache.cassandra.distributed.upgrade.MixedModeReadTest.checkTraceForDigestMismatch(MixedModeReadTest.java:89)
> at
> org.apache.cassandra.distributed.upgrade.MixedModeReadTest.lambda$mixedModeReadColumnSubsetDigestCheck$0(MixedModeReadTest.java:63)
> at
> org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:171)
> at
> org.apache.cassandra.distributed.upgrade.MixedModeReadTest.mixedModeReadColumnSubsetDigestCheck(MixedModeReadTest.java:76)
> {code}
> The initial investigation shows that
> * The test only fails in the setup phase of
> mixedModeReadColumnSubsetDigestCheck. The cluster version is
> *Versions.Major.v3X*
> * The test failure is likely a consequence of CASSANDRA-15962. After
> dropping the commit in branch cassandra-3.11 and rebuild the dtest jar, the
> upgrade test can pass. Meanwhile, dropping the other commits does not help.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
