Thanks Robert. Yup, I increased "in_memory_compaction_limit_in_mb" to 512MB so the row in question fits into it and ran repair on a couple of nodes owning its key. The log entries about this particular row went away and those columns haven't reappeared, yet. If that was the reason, that's unfortunate cause we have rows much larger than 512MB and it'd effectively mean nothing can be deleted from them... Can't increase this parameter forever.
I'm gonna go ahead and file a report at JIRA. Roman On Wed, Mar 25, 2015 at 4:11 PM, Robert Coli <rc...@eventbrite.com> wrote: > On Wed, Mar 25, 2015 at 1:57 PM, Roman Tkachenko <ro...@mailgunhq.com> > wrote: > >> Okay, so I'm positively going crazy :) >> >> Increasing gc_grace + repair + decreasing gc_grace didn't help. The >> columns still appear after the repair. I checked in cassandra-cli and >> timestamps for these columns are old, not in the future, so it shouldn't be >> the reason. >> >> I also did a test: updated one of columns and it was indeed updated. Then >> deleted it (and it was deleted), ran repair and its "updated" version >> reappeared again! Why wouldn't these columns just go away? Is there any way >> I can force their deletion permanently? >> > > It sounds like you have done enough sanity checking of your use of > Cassandra to consider filing this issue as a JIRA on the issues.apache.org > JIRA. > > The fact that it seems to only affect a row that is being compacted > incrementally is an interesting datapoint... > > =Rob > >
