Commitlog segments remain until all the data in them has been flushed. Reduce MemtableFlushAfterMinutes.
If I had to guess without your error log why the node went down, I would guess you exceeded the open file handle allowance. You can increase that with the standard ulimit or /etc/security/limits.conf methods. On Wed, Oct 6, 2010 at 1:51 PM, Rana Aich <aichr...@gmail.com> wrote: > Hello Experts, > I see a queer behavior from on of the Cassandra nodes in my cluster where > the data is not flushed off Commitlogs and the Commitlog file grows in > number. I was inserting the data into the cluster and since yesterday this > node had more than 900 commitlog files. > -rw-r--r-- 1 dev dev 134218072 2010-10-05 15:11 CommitLog-1286218976629.log > -rw-r--r-- 1 dev dev 134217909 2010-10-05 15:11 CommitLog-1286316625590.log > -rw-r--r-- 1 dev dev 134217872 2010-10-05 15:15 CommitLog-1286316673050.log > -rw-r--r-- 1 dev dev 134217957 2010-10-05 15:19 CommitLog-1286316917618.log > -rw-r--r-- 1 dev dev 134217764 2010-10-05 15:21 CommitLog-1286317178922.log > -rw-r--r-- 1 dev dev 134217871 2010-10-05 15:24 CommitLog-1286317311372.log > -rw-r--r-- 1 dev dev 134218029 2010-10-05 15:25 CommitLog-1286317446787.log > -rw-r--r-- 1 dev dev 134218065 2010-10-05 15:26 CommitLog-1286317506892.log > -rw-r--r-- 1 dev dev 134218100 2010-10-05 15:27 CommitLog-1286317567927.log > -rw-r--r-- 1 dev dev 134218020 2010-10-05 15:29 CommitLog-1286317634668.log > -rw-r--r-- 1 dev dev 134217922 2010-10-05 15:31 CommitLog-1286317741576.log > -rw-r--r-- 1 dev dev 134217944 2010-10-05 15:32 CommitLog-1286317887655.log > -rw-r--r-- 1 dev dev 134217961 2010-10-05 15:33 CommitLog-1286317968395.log > -rw-r--r-- 1 dev dev 134217926 2010-10-05 15:34 CommitLog-1286318033241.log > -rw-r--r-- 1 dev dev 134218045 2010-10-05 15:36 CommitLog-1286318097727.log > -rw-r--r-- 1 dev dev 134218027 2010-10-05 15:37 CommitLog-1286318190654.log > -rw-r--r-- 1 dev dev 134218060 2010-10-05 15:38 CommitLog-1286318277987.log > -rw-r--r-- 1 dev dev 134218055 2010-10-05 15:41 CommitLog-1286318338561.log > -rw-r--r-- 1 dev dev 134218069 2010-10-05 15:41 CommitLog-1286318461180.log > -rw-r--r-- 1 dev dev 134217791 2010-10-05 15:43 CommitLog-1286318519193.log > -rw-r--r-- 1 dev dev 134218066 2010-10-05 15:44 CommitLog-1286318609960.log > -rw-r--r-- 1 dev dev 134218093 2010-10-05 15:45 CommitLog-1286318692656.log > It seems that commitlog files spawned every minute and when it grew over 900 > files the Cassandra node went down. > Now I'm trying to start my node but it has started to read all of those > Commitlog files and takes huge amount of time to restart. > Can anyone throw some light on this unusual behavior. Till yesterday before > starting the data insertion the node had almost 550GB of data. Incidentally > I did stop another node prior to inserting the data because it was 100% > full. I'm using RandomPartitioner with Initial Tokens. > Thanking all in advance. > Rana > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
