Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data Size = 21 TB.
And this is the nodetool cfstats output for one of our busiest column families: SSTable count: 10 Space used (live): 43239294899 Space used (total): 43239419603 SSTable Compression Ratio: 0.2954468408497778 Number of Keys (estimate): 63729152 Memtable Columns Count: 1921620 Memtable Data Size: 257680020 Memtable Switch Count: 9 Read Count: 6167 Read Latency: NaN ms. Write Count: 770984 Write Latency: 0.098 ms. Pending Tasks: 0 Bloom Filter False Positives: 370 Bloom Filter False Ratio: 0.00000 Bloom Filter Space Used: 80103200 Compacted row minimum size: 180 Compacted row maximum size: 3311 Compacted row mean size: 2631 Average live cells per slice (last five minutes): 73.0 Average tombstones per slice (last five minutes): 13.0 On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <mark.re...@boxever.com> wrote: > What is your data size and number of columns in Cassandra. Do you do many > deletions? > > > On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ruchir....@gmail.com> wrote: > >> Really curious to know what's causing the spike in Columns and >> DeletedColums below : >> >> >> 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class >> Histogram: >> num #instances #bytes class name >> ---------------------------------------------- >> 1: 132626060 6366050880 java.nio.HeapByteBuffer >> 2: 28194918 3920045528 [B >> 3: 78124737 3749987376 >> edu.stanford.ppl.concurrent.SnapTreeMap$Node >> * 4: 67650128 2164804096 >> <2164804096> org.apache.cassandra.db.Column* >> * 5: 16315310 522089920 >> org.apache.cassandra.db.DeletedColumn* >> 6: 6818 392489608 [I >> 7: 2844374 273059904 >> edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch >> 8: 5727000 229080000 java.util.TreeMap$Entry >> 9: 767742 182921376 [J >> 10: 2932832 140775936 >> edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder >> 11: 2844375 91020000 >> edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch >> 12: 4145131 66322096 >> java.util.concurrent.atomic.AtomicReference >> 13: 437874 64072392 [C >> 14: 2660844 63860256 >> java.util.concurrent.ConcurrentSkipListMap$Node >> 15: 4920 62849864 [[B >> 16: 1632063 52226016 >> edu.stanford.ppl.concurrent.SnapTreeMap >> > >