Harikrishnan created CASSANDRA-12844:
----------------------------------------
Summary: nodetool drain causing mutiple nodes crashing with hint
file corruption in Cassandra 3.9
Key: CASSANDRA-12844
URL: https://issues.apache.org/jira/browse/CASSANDRA-12844
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Harikrishnan
The steps are as follows.
we have 4/4 node cassandra running in 3.9 version.
In one node made some changes to cassanra.yaml. issued a nodetool drain
killed the cassandra process and restarted the node. After sometime nodetool
status reported multiple nodes are down in that DC.
Went and check the system.log of all the files and found the hint corruption
occuring(CASSANDRA-12728). nodetool drain causing this corruption and bringing
multiple nodes down is a big concern.
ERROR [HintsDispatcher:2] 2016-10-26 12:17:59,361
HintsDispatchExecutor.java:225 - Failed to dispatch hints file
4d1362f0-053c-4042-80a7-bfc85a26c90f-1477509190999-1.hints: file is corrupted
({})
org.apache.cassandra.io.FSReadError: java.io.EOFException
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
[apache-cassandra-3.9.jar:3.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_102]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
