Hi, We have a small cluster of 3 Cassandra servers running w/ full replication. Every once in a while we get an OutOfMemory exception and have to restart servers. Sometimes just restarting doesn’t do it and we have to clean the commitlog or data directory.
We are running Cassandra 0.6.8. There is only 1 keyspace and 3 column families. There are less than 1000 keys across all column families. There is roughly 1 write request per second and 1 read request. Each server is allocated 1GB. Size of all files in data directory of the only column family is ~300MB. MemtableThroughputInMB is throttled way down to 2 and BinaryMemtableThroughputInMB to 8 (w/ higher values we were running out of memory extremely fast, this way it works for a couple of days w/o crashing). Last time this issue happened, I didn’t clear the commitlog/data folders, enabled gc logging and restarted Cassandra. It crashes really fast, but what is really strange is that it seems like it still has plenty of memory when the error happens, last 3 lines from gc log: 21.408: [GC 437098K->436592K(1046464K), 0.0986800 secs] 21.520: [GC 453616K->453117K(1046464K), 0.0967770 secs] 21.629: [GC 470141K->469436K(1046464K), 0.0383520 secs] The full log is here: http://pastebin.com/XGRSRcBd I’ve tried increasing the memory up to 1.5GB, but it still doesn’t start. Any ideas what might be the problem here? Thank you, Aram
