There's an open bug for users that have offheap memtables and secondary index - there's at least a few people reporting an error flushing that blocks future flushes.
If you're seeing that, and use that combo, you may want to switch to on-heap memtables (or contribute a patch to fix the offheap+2i interaction) On Tue, Jan 30, 2018 at 9:06 AM, Chris Lohfink <clohf...@apple.com> wrote: > The commitlog growing is often a symptom of a problem. If the memtable > flush or post flush fails in anyway, the commitlogs will not be > recycled/deleted and will continue to pool up. > > Might want to go back in logs earlier to make sure theres nothing like the > postmemtable flusher getting a permission error (some tooling creates > commitlogs so if run by wrong user can create this prooblem), or a memtable > flush error. You can also check tpstats to see if tasks are queued up in > postmemtable flusher and jstack to see where the active ones are stuck if > they are. > > Chris > > On Jan 30, 2018, at 4:20 AM, Amit Singh <dev...@nic.in> wrote: > > Hi, > > When you actually say nodetool flush, data from memTable goes to disk > based structure as SStables and side by side , commit logs segments for > that particular data get written off and its continuous process . May be in > your case , you can decrease the value of below uncommented property in > Cassandra.yaml > > commitlog_total_space_in_mb > > Also this is what is it used for > > # Total space to use for commit logs on disk. > # > # If space gets above this value, Cassandra will flush every dirty CF > # in the oldest segment and remove it. So a small total commitlog space > # will tend to cause more flush activity on less-active columnfamilies. > # > # The default value is the smaller of 8192, and 1/4 of the total space > # of the commitlog volume. > > > *From:* Mokkapati, Bhargav (Nokia - IN/Chennai) [mailto:bhargav.mokkapati@ > nokia.com <bhargav.mokkap...@nokia.com>] > *Sent:* Tuesday, January 30, 2018 4:00 PM > *To:* user@cassandra.apache.org > *Subject:* Commitlogs are filling the Full Disk space and nodes are down > > Hi Team, > > My Cassandra version : Apache Cassandra 3.0.13 > > Cassandra nodes are down due to Commitlogs are getting filled up until > full disk size. > > <image001.jpg> > > With “Nodetool flush” I didn’t see any commitlogs deleted. > > Can anyone tell me how to flush the commitlogs without losing data. > > Thanks, > Bhargav M > > >
