During performance testing being run on our 4 node Cassandra 1.1.5 cluster, we are seeing warning logs about the heap being almost full - [1]. I'm trying to figure out why, and how to prevent it.
The tests are being run on a Cassandra ring consisting of 4 dedicated boxes with 32 GB of RAM each. The heap size is set to 8 GB as recommended. All the other relevant settings I know off are the default ones: - memtable_total_space_in_mb is not set in the yaml, so should default to 1/3rd the heap size. - They key cache should be 100 MB at the most. I checked the key cache the day after the tests were run via nodetool info, and it reported 4.5 MB being used. - row cache is not being used - I summed up the bloom filter usage reported by nodetool cfstats in all the CFs and it was under 50 MB. The resident size of the cassandra process accd to top is 8.4g even now. Did a heap histogram using jmap, but not sure how to interpret those results usefully - [2]. Performance test details: - The test is write only, and is writing relatively large amount of data to one CF. - There is some other traffic that is constantly on that writes smaller amounts of data to many CFs, and does some reads. The total number of CFs are 114, but quite a few of them are not used. Thanks, Arindam [1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1: WARN GCInspector.java (line 145) Heap is 0.8287082580489245 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically [2] Object Histogram: num #instances #bytes Class description -------------------------------------------------------------------------- 1: 152855 86035312 int[] 2: 13395 45388008 long[] 3: 49517 9712000 java.lang.Object[] 4: 120094 8415560 char[] 5: 145106 6965088 java.nio.HeapByteBuffer 6: 40525 5891040 * ConstMethodKlass 7: 231258 5550192 java.lang.Long 8: 40525 5521592 * MethodKlass 9: 134574 5382960 java.math.BigInteger 10: 36692 4403040 java.net.SocksSocketImpl 11: 3741 4385048 * ConstantPoolKlass 12: 63875 3538128 * SymbolKlass 13: 104048 3329536 java.lang.String 14: 132636 3183264 org.apache.cassandra.db.DecoratedKey 15: 97466 3118912 java.util.concurrent.ConcurrentHashMap$HashEntry 16: 97216 3110912 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
