All, I need help interpreting the results of my investigation. I'm encountering this error: "Unable to reduce heap usage since there are no dirty column families". My heap sits near max and occasionally OOMs. (4GB heap)
Following Mr. Ellis's instructions here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Memory-Usage-During-Read-td6338635.html I set the heap down to 1GB, restarted the node, watched the memory climb in jconsole and waited for a heap dump. (Of course the first time I tried this I got a permission denied error on writing out the dump, and had to restart C* as root, but anyway ...) Below you'll find a screen grab of the heap dump analysis. http://screencast.com/t/U6IYzloe2McP Here is what I see in cassandra.log just prior to OOM: [ec2-user@ip-10-86-223-245 ~]$ tail -f /var/log/cassandra/cassandra.log INFO 22:37:11,193 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 22:37:11,194 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 22:37:11,195 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 22:37:11,196 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 22:37:11,212 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 22:38:32,485 Opening /cassandra/data/Keyspace1/TwitterTest-g-5852 INFO 22:38:33,253 Opening /cassandra/data/Keyspace1/TwitterTest-g-5502 INFO 22:38:34,710 Opening /cassandra/data/Keyspace1/TwitterTest-g-5643 INFO 22:38:35,653 Opening /cassandra/data/Keyspace1/TwitterTest-g-6117 INFO 22:38:35,699 Opening /cassandra/data/Keyspace1/TwitterTest-g-1376 [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor27] ...etc... >From my org.apache.cassandra.db MBean I get an estimate of 596,504,576 keys. ----- What I'd really like to know is: 1. What operation is C* performing during lines like these: INFO 22:38:34,710 Opening /cassandra/data/Keyspace1/TwitterTest-g-5643 (I think this is an SSTable it's extracting an index for this column family from) 2. Has my CF index outgrown memory? 3. If so is there a way to relate # CF, # Columns, # Rows to index size? I need to know how many keys I can store before I need more memory, or need more nodes. Thanks in advance. I've been getting a lot of help from the list and I really appreciate it! Ian
