Thanks Peter for the reply. We are currently "fixing" our inconsistent data (since we have master data saved) .
We will follow your suggestion and we will run Node Repair tool more often in the future. However, what happens to data inserted/deleted after Node Repair tool runs (i.e., between Node Repair and Major Compaction). Vram On Tue, Jan 11, 2011 at 12:33 AM, Peter Schuller <peter.schul...@infidyne.com> wrote: >> I have few more questions: >> >> 1. If we change the write/delete consistency level to ALL, do we >> eliminate the data inconsistency among nodes (since the delete >> operations will apply to ALL replicas)? >> >> 2. My understanding is that "Read Repair" doesn't handle tombstones. >> How about "Node Tool Repair" (do we still see inconsistent data among >> nodes after running "Node Tool Repair")? > > Read repair and nodetool repair handle it during normal circumstances. > The root cause here is that not running nodetool repair within > GCGraceSeconds breaks the underlying design, leading to the type of > inconsistency you got that is not healed by RR or repair. > > The most important thing is to, from now on, make sure nodetool repair > is run often enough - either by running it more often or by increasing > GCGraceSeconds - so that deletes are never forgotten to begin with. > > In terms of what to do now that you're in this position, my summary of > my understanding based on the JIRA ticket and DistributedDeletes is > here: > > http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds > > Running at CL.ALL won't help in this case since the expiry of the > tombstones means the inconsistency won't be reconciled (see the JIRA > ticket). If you're not in this position (i.e., nodetool repair having > been run often enough). Using CL.ALL could technically "help" in the > normal case, but that's not the best way to heal consistency. Instead, > let Cassandra use normal read-repair. Using CL.ALL means, for one > thing, that you cannot survive node failures since CL.ALL queries will > start failing. > > Basically, the tombstone issue is a non-problem as long as you run > nodetool repair often enough with respect to GCGraceSeconds. The > situation right now is a bit special because the contraints of the > cluster were violated (i.e., expired tombstones prior to nodetool > repair having been run). > > I hope that clarifies. > > -- > / Peter Schuller >
