Thanks Dean. I will try the node drain next, however Do you know if this is a known issue/bug with 1.1, I scanned through some 200 odd jira entries that have commit log in the text for some clues -but no luck.
Amit On Tue, Mar 12, 2013 at 12:17 PM, Hiller, Dean <dean.hil...@nrel.gov> wrote: > Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB > [root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog > a5: 1.1G /opt/datastore/commitlog > a3: 1.1G /opt/datastore/commitlog > a1: 1.1G /opt/datastore/commitlog > a2: 1006M /opt/datastore/commitlog > a4: 1.1G /opt/datastore/commitlog > a6: 1.1G /opt/datastore/commitlog > > If you run nodetool drain on a node, you can wipe the commit logs after > that(we QA tested this but don't take my word for it). We also found out > drain was moving data to the sstables but did not seem to delete from the > commit log at all as commit log space used remained the same after a drain. > We did not fully test removing the commit log files so you should try to > do that yourself with a test in QA. > > Later, > Dean > > From: a k <kumaramit.ex...@gmail.com<mailto:kumaramit.ex...@gmail.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Date: Tuesday, March 12, 2013 10:46 AM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: commitlog -deleted keyspaces. > > We are running a 4 node cluster version 1.1.0 and our commit logs seem to > be ever growing. We have a total about 250 GB per node in the > keyspaces/column families and the commit logs are at about 30 GB.There have > been several deletions of keyspaces in our setup and I am concerned about a > few things. > > First being the size of the commit logs, we have not modified the > "commitlog_total_space_in_mb" in yaml so I assume it is the default > 4092MB, We don't have "memtable_flush_after_mins" setting either. > > Is this sane? Why would the size of the commit logs be so large (I am > worried about the startup time, when it has to replay the commit logs) and > why would the default size of 4092 MB not be enforced. > > Would it cause us trouble when we upgrade to 1.2? > > Another thing I have noticed is that upon restarts, the old keyspaces that > were deleted re-appear although with less data, I would imagine that has > nothing to do with the commit logs. > > Can I safely delete the commitlogs after the nodetool flush? > > > > Many thanks > Amit > >
