figured out the problems, node 1 has configuration error due to a problem
in ansible script, which set rpc_address to localhost.
On node 3, after clearing the logs and restarting I saw this in log file
INFO [main] 2017-07-26 09:26:36,938 Server.java:160 - Starting
listening for CQL clients on /10.128.1.3:9042 (unencrypted)...
INFO [main] 2017-07-26 09:26:36,994 ThriftServer.java:119 - Binding
thrift service to /10.128.1.3:9160
INFO [Thread-3] 2017-07-26 09:26:36,999 ThriftServer.java:136 -
Listening for thrift clients...
ERROR [HintsDispatcher:1] 2017-07-26 09:28:30,275
HintsDispatchExecutor.java:225 - *Failed to dispatch hints file
702e8a31-6441-4444-b569-d2d137d54a5d-1500589737308-1.hints: file is
corrupted ({})*
org.apache.cassandra.io.FSReadError: java.io.EOFException: EOF after
736 bytes out of 1955
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
[apache-cassandra-3.9.jar:3.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.io.EOFException: EOF after 736 bytes out of 1955
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.ChecksummedDataInput.readFully(ChecksummedDataInput.java:153)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.readBuffer(HintsReader.java:312)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:303)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
~[apache-cassandra-3.9.jar:3.9]
... 15 common frames omitted
ERROR [HintsDispatcher:1] 2017-07-26 09:28:30,276
CassandraDaemon.java:226 - Exception in thread
Thread[HintsDispatcher:1,1,main]
org.apache.cassandra.io.FSReadError: java.io.EOFException: EOF after
736 bytes out of 1955
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
~[apache-cassandra-3.9.jar:3.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.io.EOFException: EOF after 736 bytes out of 1955
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.ChecksummedDataInput.readFully(ChecksummedDataInput.java:153)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.readBuffer(HintsReader.java:312)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:303)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
~[apache-cassandra-3.9.jar:3.9]
... 15 common frames omitted
ERROR [HintsDispatcher:1] 2017-07-26 09:28:30,278
StorageService.java:392 - Stopping gossiper
WARN [HintsDispatcher:1] 2017-07-26 09:28:30,278
StorageService.java:299 - Stopping gossip by operator request
INFO [HintsDispatcher:1] 2017-07-26 09:28:30,278 Gossiper.java:1500 -
Announcing shutdown
INFO [HintsDispatcher:1] 2017-07-26 09:28:30,280
StorageService.java:2087 - Node /10.128.1.3 state jump to shutdown
ERROR [HintsDispatcher:1] 2017-07-26 09:28:32,281
StorageService.java:397 - Stopping RPC server
INFO [HintsDispatcher:1] 2017-07-26 09:28:32,281
ThriftServer.java:142 - Stop listening to thrift clients
ERROR [HintsDispatcher:1] 2017-07-26 09:28:32,282
StorageService.java:402 - Stopping native transport
INFO [HintsDispatcher:1] 2017-07-26 09:28:32,284 Server.java:180 -
Stop listening for CQL clients
one of the hint file was corrupted, which is a known bug
https://issues.apache.org/jira/browse/CASSANDRA-12728
it was resolved in 3.0.14 and 3.11.0 version.
removing the corrupt file solved the issue for me
On Wed, Jul 26, 2017 at 10:30 AM, Junaid Nasir <jna...@an10.io> wrote:
> each node has its own ip for listen_address and rpc_address. seed node ip
> is fixed to 10.128.1.1 on all nodes. configuration was written using
> ansible and I have also verified it.
>
> On Wed, Jul 26, 2017 at 3:52 AM, Erik Forkalsud <eforkals...@cj.com>
> wrote:
>
>> On 07/25/2017 05:13 AM, Junaid Nasir wrote:
>>
>> listen_address: 10.128.1.1
>>
>> rpc_address: 10.128.1.1
>>
>> Are these the values on all three nodes?
>>
>> If so, try with empty values:
>>
>> listen_address:
>>
>> rpc_address:
>>
>> or make sure each node has its own IP address configured.
>>
>>
>>
>
