I see about 3000 lines of, INFO [COMMIT-LOG-WRITER] 2010-06-23 16:40:29,107 CommitLog.java (line 412) Discarding obsolete commit log:CommitLogSegment(/data1/cass/commitlog/CommitLog-1277302220723.log)
Then, http://pastebin.com/YQA0mpRG It's around 16:50 that cassandra writes stop timing out. Some writes are getting through during this 10 minutes, but they shouldn't be enough to cause the index memtables to flush. Thanks, Sean On Wed, Jun 23, 2010 at 3:30 PM, Benjamin Black <b...@b3k.us> wrote: > Are you seeing any sort of log messages from Cassandra at all? > > On Wed, Jun 23, 2010 at 2:26 PM, Sean Bridges <sean.brid...@gmail.com> wrote: >> We were running a load test against a single 0.6.2 cassandra node. 24 >> hours into the test, Cassandra appeared to be nearly frozen for 10 >> minutes. Our write rate went to almost 0, and we had a large number >> of write timeouts. We weren't swapping or gc'ing at the time. >> >> It looks like the problems were caused by our memtables flushing after >> 24 hours (we have MemtableFlushAfterMinutes=1440). Some of our column >> families are written to infrequently so that they don't hit the flush >> thresholds in MemtableOperationsInMillions and MemtableThroughputInMB. >> After 24 hours we had ~3000 commit log files. >> >> Is this flushing causing Cassandra to become unresponsive? I would >> have thought Cassandra could flush in the background without blocking >> new writes. >> >> Thanks, >> >> Sean >> >
