I've been searching on wikis and FAQs for a definitive answer to this and haven't found it yet so I thought I'd ask people here.
We have a 5-node cluster set up with a replication factor of 3. We're doing write operations (using batch mutates that include deletions) with a QUORUM consistency level. For sake of discussion, let's say that the nodes are always up and fully operational and responsive. When we do a delete on a column in the above configuration, the call to the server won't return until 2 of the 3 replicas are written to unless there's an error. That part is well-documented and understood. The question I have is whether or not the last of the 3 replica nodes gets the delete request (which would result in a tombstone being added to it). Does the code finish up writing to the rest of the replicas in a background thread? What I've read implies that the 3 replicas will get the delete request but it's not explicit. If there's no guarantee that the 3rd replica will get it (assuming that there aren't node problems that would preclude it from receiving the delete request), then that implies to me that I should follow each delete with a read so a ReadRepair will pick up the discrepancy and fix it (or do a nodetool repair). Or am I missing something? An underlying question is if anything special needs to be done when nothing "out of the ordinary" happens to the cluster (e.g., node goes down, network problems, etc.) in order to have eventual consistency. Thanks, Scott
