1.0 branch is less stable then 0.8 for production. We discovered
following problems:
1. memory leak in scrub (also reported on this list)
2. problem with saving key caches for super column family - CASSANDRA-3511
3. in 1.0.3 some hints are stuck in system tables. Hints to other nodes
seems to deliver fine, but node is probably writing also hints to itself
and they get stuck there. I cleaned hints in entire cluster (stopped it)
to be sure that no < 1.0 hints are there
4. memory leak. Node boots and work fine, but after some time (few
hours) it is doing stuff like:
WARN [ScheduledTasks:1] 2011-11-22 07:48:18,843 GCInspector.java (line
146) Heap is 0.8598437159194559 full. You may need to reduce memtable
and/or cache sizes. Cassandra will now flush up to the two largest
memtables to free up memory. Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
WARN [ScheduledTasks:1] 2011-11-22 07:48:18,844 StorageService.java
(line 2511) Flushing CFS to relieve memory pressure
INFO [ScheduledTasks:1] 2011-11-22 07:48:18,844 ColumnFamilyStore.java
(line 688) Enqueuing flush 1034292896(3645/57056 serialized/live bytes,
51 ops)
flushed table is very small. After node restart it is flushing CFs in
10MB serialized chunks. It is not caches are too large problem because
row caches are minimal (now are even reduced to 0.6 their sizes). It
works with larger caches after node restart just fine.
It will OOM because situation is getting worse. I am now monitoring this
issue more closely, collecting heap dumps.
799686 instances of class org.apache.cassandra.db.DecoratedKey
796969 instances of class org.apache.cassandra.dht.BigIntegerToken
612414 instances of class
org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
444125 instances of class org.apache.cassandra.db.ExpiringColumn
187205 instances of class
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
187205 instances of class
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue
186347 instances of class org.apache.cassandra.utils.Pair
111813 instances of class
org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
111813 instances of class org.apache.cassandra.db.ThreadSafeSortedColumns
110932 instances of class org.apache.cassandra.db.SuperColumn
