Does it always pick the node with the lowest IP address? All of my hosts are in the same /24. The fourth node in the 5 node cluster has the lowest value in the 4th octet (54). I erased the cluster and rebuilt it from scratch as a 3 node cluster using the first 3 nodes, and now the ReplicateOnWrites are all going to the third node, which is also the lowest valued IP address (55).
That would explain why only 1 node gets writes in a 3 node cluster (RF=3) and why 3 nodes get writes in a 5 node cluster, and why one of those 3 is taking 66% of the writes. > >> On another note, on a 5-node cluster, I'm only seeing 3 nodes with >> ReplicateOnWrite Completed tasks in nodetool tpstats output. Is that >> normal? I'm using RandomPartitioner... >> >> Address DC Rack Status State Load Owns >> Token >> >> 136112946768375385385349842972707284580 >> 10.0.0.57 datacenter1 rack1 Up Normal 2.26 GB 20.00% 0 >> 10.0.0.56 datacenter1 rack1 Up Normal 2.47 GB 20.00% >> 34028236692093846346337460743176821145 >> 10.0.0.55 datacenter1 rack1 Up Normal 2.52 GB 20.00% >> 68056473384187692692674921486353642290 >> 10.0.0.54 datacenter1 rack1 Up Normal 950.97 MB 20.00% >> 102084710076281539039012382229530463435 >> 10.0.0.72 datacenter1 rack1 Up Normal 383.25 MB 20.00% >> 136112946768375385385349842972707284580 >> >> The nodes with ReplicateOnWrites are the 3 in the middle. The first node >> and last node both have a count of 0. This is a clean cluster, and I've >> been doing 3k ... 2.5k (decaying performance) inserts/sec for the last 12 >> hours. The last time this test ran, it went all the way down to 500 >> inserts/sec before I killed it. > > Could be due to https://issues.apache.org/jira//browse/CASSANDRA-2890. > > -- > Sylvain
