In 1.0.0, you have: # Total space to use for commitlogs. # If space gets above this value (it will round up to the next nearest # segment multiple), Cassandra will flush every dirty CF in the oldest # segment and remove it. # commitlog_total_space_in_mb: 4096
In 0.8, you're supposed to use the memtableFlushAfterMins property for each CF to avoid filling up your commit log partition. Which is a little more involved, but that is why we have improved that in 1.0. -- Sylvain On Fri, Sep 23, 2011 at 7:47 AM, Yang <teddyyyy...@gmail.com> wrote: > thanks for the input. > > if that's the case, I think the solution would be to sort the CFs to > flush by a more complex criteria than just size. for example the > number of dirty commit logs that contain this CF should be considered > as a score. > > Yang > > On Thu, Sep 22, 2011 at 10:40 PM, Philippe <watche...@gmail.com> wrote: >> It sure looks like what I'm seeing on my cluster where a 100G commit lot >> partition fills up in 12 hours (0.8.x) >> >> Le 23 sept. 2011 03:45, "Yang" <teddyyyy...@gmail.com> a écrit : >>> in 1.0.0 we don't have memtable_throughput for each individual CF , >>> and instead >>> which memtable/CF to flush is determined by "largest >>> getTotalMemtableLiveSize() ". >>> (MeteredFlusher.java line 81) >>> >>> >>> what would happen in the following case ? : I have only 2 CF, the >>> traffic for one CF is 1000 times that >>> of the second CF, >>> so the high-traffic CF constantly triggers total mem threshold , and >>> every time, the busy CF is flushed. >>> >>> but the light-traffic CF is never flushed ( well, until we have >>> flushed about 1000 times the busy CF), >>> now we are left with many commit logs , each of them containing a few >>> entries for the light-traffic table. we have to keep these commit logs >>> because these entries are not flushed to sstable yet. >>> >>> then there are 2 problems: 1) to persist the few records from the >>> light-traffic CF, you have to keep 1000 times the commit logs >>> necessary, taking up disk space 2) when you do a recover on server >>> restart, you'll have to read through all those commit logs . >>> >>> does the above hypothesis sound right? >>> >>> Thanks >>> Yang >> >
