It might be this https://issues.apache.org/jira/browse/CASSANDRA-4201
As dean said, if you drain the node during shut down you can delete the logs. If you are paranoid just move them out of the way. > Another thing I have noticed is that upon restarts, the old keyspaces that > were deleted re-appear although with less data, I would imagine that has > nothing to do with the commit logs. Yes. Sounds like the schema change is being replayed, but to not the deletion. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 13/03/2013, at 5:34 AM, "Hiller, Dean" <[email protected]> wrote: > I do not know if it is a known issue. I only know what I saw in QA. If I > remember correctly, I tested this on 1.1.4 and 1.2.2 with the same result > that commit logs are left there(maybe it is by design….I really don't know). > All I know is that the sstables do correctly grow in size. > > Dean > > From: a k <[email protected]<mailto:[email protected]>> > Reply-To: "[email protected]<mailto:[email protected]>" > <[email protected]<mailto:[email protected]>> > Date: Tuesday, March 12, 2013 3:54 PM > To: "[email protected]<mailto:[email protected]>" > <[email protected]<mailto:[email protected]>> > Subject: Re: commitlog -deleted keyspaces. > > Thanks Dean. I will try the node drain next, however Do you know if this is a > known issue/bug with 1.1, I scanned through some 200 odd jira entries that > have commit log in the text for some clues -but no luck. > > Amit > > > On Tue, Mar 12, 2013 at 12:17 PM, Hiller, Dean > <[email protected]<mailto:[email protected]>> wrote: > Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB > [root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog > a5: 1.1G /opt/datastore/commitlog > a3: 1.1G /opt/datastore/commitlog > a1: 1.1G /opt/datastore/commitlog > a2: 1006M /opt/datastore/commitlog > a4: 1.1G /opt/datastore/commitlog > a6: 1.1G /opt/datastore/commitlog > > If you run nodetool drain on a node, you can wipe the commit logs after > that(we QA tested this but don't take my word for it). We also found out > drain was moving data to the sstables but did not seem to delete from the > commit log at all as commit log space used remained the same after a drain. > We did not fully test removing the commit log files so you should try to do > that yourself with a test in QA. > > Later, > Dean > > From: a k > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Reply-To: > "[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" > > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Date: Tuesday, March 12, 2013 10:46 AM > To: > "[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" > > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Subject: commitlog -deleted keyspaces. > > We are running a 4 node cluster version 1.1.0 and our commit logs seem to be > ever growing. We have a total about 250 GB per node in the keyspaces/column > families and the commit logs are at about 30 GB.There have been several > deletions of keyspaces in our setup and I am concerned about a few things. > > First being the size of the commit logs, we have not modified the > "commitlog_total_space_in_mb" in yaml so I assume it is the default 4092MB, > We don't have "memtable_flush_after_mins" setting either. > > Is this sane? Why would the size of the commit logs be so large (I am worried > about the startup time, when it has to replay the commit logs) and why would > the default size of 4092 MB not be enforced. > > Would it cause us trouble when we upgrade to 1.2? > > Another thing I have noticed is that upon restarts, the old keyspaces that > were deleted re-appear although with less data, I would imagine that has > nothing to do with the commit logs. > > Can I safely delete the commitlogs after the nodetool flush? > > > > Many thanks > Amit > >
