Hello all!
Thanks for taking the time to read this, I'm new to Cassandra and really
want to get it working :)
Below you'll find the output from tpstats on the three nodes in my .8.1
cluster. As you'll note from ring (also below), my cluster appears balanced.
However, please note that NODE1 is the only node that has completed any
ReplicateOnWriteStage actions. I've been having a lot of trouble with RoW
and seen a single node in my various test clusters get hugely backed up on
this stage to the point where it slows all writes to a crawl.
I'm using counters in a column family defined like this:
==== KEYSPACE/CF ====
create keyspace Keyspace1
with strategy_options=[{replication_factor:3}]
and placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy';
use Keyspace1;
create column family TwitterTest
with default_validation_class = CounterColumnType
and replicate_on_write = true;
====
With any given batch_insert() I'm incrementing ~17,000 counters. I've backed
off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
elsewhere that ANY or ONE could overwhelm the system.
I hope I'm just doing something stupid and one you experienced folks can
point out my error.
Thanks in advance!
Ian
==== MORE DATA ====
Address DC Rack Status State Load Owns
Token
113427455640312821154458202477256070484
10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
33.33% 0
10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
33.33% 56713727820156410577229101238628035242
10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
33.33% 113427455640312821154458202477256070484
SEED
Pool Name Active Pending Completed
ReadStage 0 0 7
RequestResponseStage 0 0 1479839
MutationStage 0 0 1479908
ReadRepairStage 0 0 0
ReplicateOnWriteStage 0 0 0
GossipStage 0 0 5176
AntiEntropyStage 0 0 0
MigrationStage 0 0 128
MemtablePostFlusher 0 0 28
StreamStage 0 0 0
FlushWriter 0 0 28
MiscStage 0 0 2
FlushSorter 0 0 0
InternalResponseStage 0 0 363
HintedHandoff 0 0 0
NODE1
Pool Name Active Pending Completed
ReadStage 0 0 7
RequestResponseStage 0 0 2960208
MutationStage 0 0 1480104
ReadRepairStage 0 0 0
ReplicateOnWriteStage 0 0 1480104
GossipStage 0 0 4151
AntiEntropyStage 0 0 0
MigrationStage 0 0 128
MemtablePostFlusher 0 0 31
StreamStage 0 0 8
FlushWriter 0 0 31
MiscStage 0 0 8
FlushSorter 0 0 0
InternalResponseStage 0 0 1
HintedHandoff 0 0 0
NODE2
Pool Name Active Pending Completed
ReadStage 0 0 7
RequestResponseStage 0 0 0
MutationStage 0 0 1481530
ReadRepairStage 0 0 0
ReplicateOnWriteStage 0 0 0
GossipStage 0 0 4201
AntiEntropyStage 0 0 0
MigrationStage 0 0 128
MemtablePostFlusher 0 0 31
StreamStage 0 0 6
FlushWriter 0 0 31
MiscStage 0 0 6
FlushSorter 0 0 0
InternalResponseStage 0 0 1
HintedHandoff 0 0 0
