Hi,
We have upgraded Cassandra from 2.0.9 to 2.1.11 in our loadtest environment
with pretty much same yaml settings in both (removed unused yaml settings and
renamed few others) and we have noticed performance on 2.1.11 is worse compared
to 2.0.9. After more investigation we found that the performance gets worse as
we increase replication factor on 2.1.11 where as on 2.0.9 performance is more
or less same. Has anything architecturally changed as far as replication is
concerned in 2.1.11?
All googling only suggested 2.1.11 should be FASTER than 2.0.9 so we are
obviously doing something different. However the client code, load test is all
identical in both cases.
Details:
Nodes: 3 ec2 c3.2x large
R/W Consistency: QUORUM
Renamed memtable_total_space_in_mb to memtable_heap_space_in_mb and removed
unused properties from yaml file.
We run compaction aggressive compaction with low gc_grace (15 mins) but this is
true for both 2.0.9 and 2.1.11.
As you can see, all p50, p90 and p99 latencies stayed with in 10% difference on
2.0.9 when we increased RF from 1 to 3, where as on 2.1.11 latencies almost
doubled (especially reads are much slower than writes).
# Nodes RF # of rows 2.0.9 2.1.11
READ
P50 P90 P99 P50 P90 P99
3 1 450 306 594 747 425 849 1085
3 3 450 358 634 877 708 1274 2642
WRITE
3 1 10 26 80 179 37 131 196
3 3 10 31 96 184 46 166 468
Any pointers on how to debug performance issues will be appreciated.
Praveen
