I ran stress test to read 50K rows and since then I am getting below error
even though ring show all nodes are up:
ERROR 12:40:29,999 Exception:
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked
down. Retry burden pushed out to client.
at
me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:308)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:213)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:100)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:106)
at
me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:203)
at
me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:200)
at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:200)
at
com.riptano.cassandra.stress.InsertCommand.call(InsertCommand.java:117)
at
com.riptano.cassandra.stress.InsertCommand.call(InsertCommand.java:1)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
---
No errors logged in the system.log and tpsats shows nothing.
nodetool -h dsdb1 tpstats
Pool Name Active Pending Completed
ReadStage 0 0 50176
RequestResponseStage 0 0 207223
MutationStage 0 0 199473
ReadRepairStage 0 0 14615
GossipStage 0 0 39835
AntiEntropyStage 0 0 0
MigrationStage 0 0 207
MemtablePostFlusher 0 0 386
StreamStage 0 0 0
FlushWriter 0 0 385
FILEUTILS-DELETE-POOL 0 0 1446
MiscStage 0 0 0
FlushSorter 0 0 0
InternalResponseStage 0 0 1230
HintedHandoff 0 0 7
compaction stats say pending: 0
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/All-nodes-down-even-though-ring-shows-up-tp6274152p6274152.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
