It might be this https://issues.apache.org/jira/browse/CASSANDRA-4201
As dean said, if you drain the node during shut down you can delete the logs. If you are paranoid just move them out of the way. > Another thing I have noticed is that upon restarts, the old keyspaces that > were deleted re-appear although with less data, I would imagine that has > nothing to do with the commit logs. Yes. Sounds like the schema change is being replayed, but to not the deletion. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 13/03/2013, at 5:34 AM, "Hiller, Dean" <dean.hil...@nrel.gov> wrote: > I do not know if it is a known issue. I only know what I saw in QA. If I > remember correctly, I tested this on 1.1.4 and 1.2.2 with the same result > that commit logs are left there(maybe it is by design….I really don't know). > All I know is that the sstables do correctly grow in size. > > Dean > > From: a k <kumaramit.ex...@gmail.com<mailto:kumaramit.ex...@gmail.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Date: Tuesday, March 12, 2013 3:54 PM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: Re: commitlog -deleted keyspaces. > > Thanks Dean. I will try the node drain next, however Do you know if this is a > known issue/bug with 1.1, I scanned through some 200 odd jira entries that > have commit log in the text for some clues -but no luck. > > Amit > > > On Tue, Mar 12, 2013 at 12:17 PM, Hiller, Dean > <dean.hil...@nrel.gov<mailto:dean.hil...@nrel.gov>> wrote: > Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB > [root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog > a5: 1.1G /opt/datastore/commitlog > a3: 1.1G /opt/datastore/commitlog > a1: 1.1G /opt/datastore/commitlog > a2: 1006M /opt/datastore/commitlog > a4: 1.1G /opt/datastore/commitlog > a6: 1.1G /opt/datastore/commitlog > > If you run nodetool drain on a node, you can wipe the commit logs after > that(we QA tested this but don't take my word for it). We also found out > drain was moving data to the sstables but did not seem to delete from the > commit log at all as commit log space used remained the same after a drain. > We did not fully test removing the commit log files so you should try to do > that yourself with a test in QA. > > Later, > Dean > > From: a k > <kumaramit.ex...@gmail.com<mailto:kumaramit.ex...@gmail.com><mailto:kumaramit.ex...@gmail.com<mailto:kumaramit.ex...@gmail.com>>> > Reply-To: > "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" > > <user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> > Date: Tuesday, March 12, 2013 10:46 AM > To: > "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" > > <user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> > Subject: commitlog -deleted keyspaces. > > We are running a 4 node cluster version 1.1.0 and our commit logs seem to be > ever growing. We have a total about 250 GB per node in the keyspaces/column > families and the commit logs are at about 30 GB.There have been several > deletions of keyspaces in our setup and I am concerned about a few things. > > First being the size of the commit logs, we have not modified the > "commitlog_total_space_in_mb" in yaml so I assume it is the default 4092MB, > We don't have "memtable_flush_after_mins" setting either. > > Is this sane? Why would the size of the commit logs be so large (I am worried > about the startup time, when it has to replay the commit logs) and why would > the default size of 4092 MB not be enforced. > > Would it cause us trouble when we upgrade to 1.2? > > Another thing I have noticed is that upon restarts, the old keyspaces that > were deleted re-appear although with less data, I would imagine that has > nothing to do with the commit logs. > > Can I safely delete the commitlogs after the nodetool flush? > > > > Many thanks > Amit > >
