I have a 12 node cassandra cluster running with RF=3. I have severl clients ( all running on a single node ) connecting to the cluster ( fixed client - node mapping ) and try to do a insert, update , select and delete. Each client has a fixed mapping of the row-keys and always connect to the same node. The timestamp on the client node is used for all operations. All operations are done using CL QUORUM.
When I run a tpstats I see the ReadRepair count consistently increasing. i need to figure out why ReadRepair is happening.. One scenario I can think of is, it could happen when there is a delay in updating the nodes to reach eventual consistency.. Let's say I have 3 nodes (RF=3) A,B,C. I insert <key> with timestamp <ts1> to A and the call will return as soon as it inserts the record to A and B. At some later point this information is sent to C... A while later A,B,C have the same data with the same timestamp. A <key,ts1> B <key, ts1> and C <key, ts1> When I update <key> on A with timestamp <ts2> to A, the call will return as soon as it inserts the record to A and B. Now the data is A <key,ts2> B <key,ts2> C <key,ts1> Assuming I query for <key> A,C respond and since there is no QUORUM, it waits for B to respond and when A,B match, the response is returned to the client and ReadRepair is sent to C. This could happen only when C is running behind in catching up the updates to A,B. Are there any stats that would let me know if the system is in a consistent state? thanks Ramesh tpstats_2011-10-05_12:50:01:ReadRepairStage 0 0 43569781 0 0 tpstats_2011-10-05_12:55:01:ReadRepairStage 0 0 43646420 0 0 tpstats_2011-10-05_13:00:02:ReadRepairStage 0 0 43725850 0 0 tpstats_2011-10-05_13:05:01:ReadRepairStage 0 0 43790047 0 0 tpstats_2011-10-05_13:10:02:ReadRepairStage 0 0 43869704 0 0 tpstats_2011-10-05_13:15:01:ReadRepairStage 0 0 43945635 0 0 tpstats_2011-10-05_13:20:01:ReadRepairStage 0 0 44020406 0 0 tpstats_2011-10-05_13:25:02:ReadRepairStage 0 0 44093227 0 0 tpstats_2011-10-05_13:30:01:ReadRepairStage 0 0 44167455 0 0 tpstats_2011-10-05_13:35:02:ReadRepairStage 0 0 44247519 0 0 tpstats_2011-10-05_13:40:01:ReadRepairStage 0 0 44312726 0 0 tpstats_2011-10-05_13:45:01:ReadRepairStage 0 0 44387633 0 0 tpstats_2011-10-05_13:50:01:ReadRepairStage 0 0 44443683 0 0 tpstats_2011-10-05_13:55:02:ReadRepairStage 0 0 44499487 0 0 tpstats_2011-10-05_14:00:01:ReadRepairStage 0 0 44578656 0 0 tpstats_2011-10-05_14:05:01:ReadRepairStage 0 0 44647555 0 0 tpstats_2011-10-05_14:10:02:ReadRepairStage 0 0 44716730 0 0 tpstats_2011-10-05_14:15:01:ReadRepairStage 0 0 44776644 0 0 tpstats_2011-10-05_14:20:01:ReadRepairStage 0 0 44840237 0 0 tpstats_2011-10-05_14:25:01:ReadRepairStage 0 0 44891444 0 0 tpstats_2011-10-05_14:30:01:ReadRepairStage 0 0 44931105 0 0 tpstats_2011-10-05_14:35:02:ReadRepairStage 0 0 44976801 0 0 tpstats_2011-10-05_14:40:01:ReadRepairStage 0 0 45042220 0 0 tpstats_2011-10-05_14:45:01:ReadRepairStage 0 0 45112141 0 0 tpstats_2011-10-05_14:50:02:ReadRepairStage 0 0 45177816 0 0 tpstats_2011-10-05_14:55:02:ReadRepairStage 0 0 45246675 0 0 tpstats_2011-10-05_15:00:01:ReadRepairStage 0 0 45309533 0 0 tpstats_2011-10-05_15:05:01:ReadRepairStage 0 0 45357575 0 0 tpstats_2011-10-05_15:10:01:ReadRepairStage 0 0 45405943 0 0 tpstats_2011-10-05_15:15:01:ReadRepairStage 0 0 45458435 0 0 tpstats_2011-10-05_15:20:01:ReadRepairStage 0 2 45508253 0 0 tpstats_2011-10-05_15:25:01:ReadRepairStage 0 0 45570375 0 0 tpstats_2011-10-05_15:30:01:ReadRepairStage 0 0 45628426 0 0 tpstats_2011-10-05_15:35:01:ReadRepairStage 0 0 45688694 0 0 tpstats_2011-10-05_15:40:01:ReadRepairStage 0 3 45743029 0 0 tpstats_2011-10-05_15:45:02:ReadRepairStage 0 0 45801167 0 0 tpstats_2011-10-05_15:50:02:ReadRepairStage 0 0 45837329 0 0 tpstats_2011-10-05_15:55:01:ReadRepairStage 0 0 45890326 0 0 tpstats_2011-10-05_16:00:01:ReadRepairStage 0 0 45951703 0 0 tpstats_2011-10-05_16:05:02:ReadRepairStage 0 0 46010736 0 0 tpstats_2011-10-05_16:10:01:ReadRepairStage 0 0 46063294 0 0 tpstats_2011-10-05_16:15:01:ReadRepairStage 0 0 46108327 0 0 tpstats_2011-10-05_16:20:01:ReadRepairStage 0 0 46142291 0 0
