*Background*: My application is running on a cluster system(which have 4 nodes), and system time of these four nodes are synchronizing by NTP. I use Write.QUORUM and Read.QUORUM strategy. The probability of this problem is not very high. Cassandra version is 1.0.3, I have tried Cassandra 1.1.1, this problem is still exist.
*Problem*: I deleted a column, but after 6 seconds, Cassandra can still get the old record which "isMarkedForDelete" is still false. Is anybody meet the same problem? And how to solve it? *Detail*: See the log below: Node 3(Local node): [pool-2-thread-42] 2012-06-27 *14:49:23,732* SliceQueryFilter.java (line 123) collecting 0 of 2147483647: SuperColumn(667072 [...... 7fffffffffffffff000001382ca96c8b636b698a:*false*:36@1340779097312016,......) [pool-2-thread-44] 2012-06-27 *14:51:21,367* StorageProxy.java (line 172) Mutations/ConsistencyLevel are [RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [ 7fffffffffffffff000001382ca96c8b636b698a:*true*:4@1340779881338000 ,]),])])]/QUORUM -- I delete this record at 14:51:21,367 [pool-2-thread-37] 2012-06-27 *14:51:27,400* SliceQueryFilter.java (line 123) collecting 0 of 2147483647: SuperColumn(667072 [......, 7fffffffffffffff000001382ca96c8b636b698a:*false*:36@1340779097312016,......) -- But I can still get the old record at 14:51:27,400 Node2: [MutationStage:118] 2012-06-27 *14:51:21,373* RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [ 7fffffffffffffff000001382ca96c8b636b698a:*true*:4@1340779881338000,]),])]) [MutationStage:118] 2012-06-27 *14:51:21,374* RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [ 7fffffffffffffff000001382ca96c8b636b698a:*true*:4@1340779881338000,]),])]) applied. Sending response to 6692098@/192.168.0.3 [MutationStage:123] 2012-06-27 14:51:27,405 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [......, 7fffffffffffffff000001382ca96c8b636b698a:*false*:36@1340779097312016 ,......]) [MutationStage:123] 2012-06-27 *14:51:27,405* RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [......, 7fffffffffffffff000001382ca96c8b636b698a:*false*:36@1340779097312016,.......]),])]) applied. Sending response to 6698516@/192.168.0.3 Node1: [MutationStage:98] 2012-06-27 *14:51:24,661* RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [ 7fffffffffffffff000001382ca96c8b636b698a:*true*:4@1340779881338000,]),])]) [MutationStage:98] 2012-06-27 *14:51:24,675* RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [ 7fffffffffffffff000001382ca96c8b636b698a: *true* :4@1340779881338000,]),])]) applied. Sending response to 6692099@/192.168.0.3 [MutationStage:93] 2012-06-27 *14:51:40,932* RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [ 7fffffffffffffff000001382ca96c8b636b698a:*true*:4@1340779900915004,]),])]) DEBUG [MutationStage:93] 2012-06-27 *14:51:40,933* RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [ 7fffffffffffffff000001382ca96c8b636b698a: *true* :4@1340779900915004,]),])]) applied. Sending response to 6706555@/192.168.0.3 [ReadStage:55] 2012-06-27 *14:51:43,074* SliceQueryFilter.java (line 123) collecting 0 of 5000:7fffffffffffffff000001382ca96c8b636b698a:*true* :4@1340779900915004 Node 4: There is no log about this record on Node 4.
