[
https://issues.apache.org/jira/browse/CASSANDRA-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280105#comment-13280105
]
Richard Low commented on CASSANDRA-3882:
----------------------------------------
I have observed this problem too, and have implemented and tested Jonathan's
suggestion of processing the response asynchronously. I have a simple
reproduction that works about 50% of the time:
1. Create a two node cluster (nodes A and B)
2. Create a keyspace
3. Take down A and wipe its data and commit log directories
4. Bring up A
5. A never learns about the keyspace
The issue is that when A restarts, it sends a MIGRATION_REQUEST message to B.
But B also sends the message to A at about the same time. Since the processing
of this is done synchronously in MigrationTask, the single thread on the
MigrationStage is blocked on both A and B. The messages timeout and retry, and
after 30 seconds it gives up.
The simple solution is to process the response asynchronously - the patch
CASSANDRA-3882-async.patch makes this change. With this, node A always learns
about the new keyspace very quickly after restarting.
As a general rule, only async processing should be done on stages, otherwise
distributed deadlocks are very likely.
Peter, is this the same issue you were seeing?
I can't think of anything that should break doing this asynchronously -
DefsTable.mergeSchema is static synchronized so it looks safe.
> avoid distributed deadlock in migration stage
> ---------------------------------------------
>
> Key: CASSANDRA-3882
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3882
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3882-hack.txt
>
>
> This is follow-up work for the remainders of CASSANDRA-3832 which was only a
> partial fix. The deadlock in the migration stage needs to be fixed, as it can
> cause bootstrap (at least) to take potentially a very very long time to
> complete, and might also cause a lack of schema propagation until otherwise
> "poked".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
