1) Is that heap dump after full GC? 2) Are you doing a lot of concurrent writes to the same partition 3) Is the system hinting at the time of the spike 4) when you say “spike” do you mean this is unusually high?
On Jul 28, 2014, at 11:07 AM, Keith Wright <kwri...@nanigans.com> wrote: > What’s your cfhistograms look like? > > From: Ruchir Jha <ruchir....@gmail.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Monday, July 28, 2014 at 10:43 AM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: Full GC in cassandra > > Also we do subsequent updates (atleat 4) for each piece of data that we write. > > > On Mon, Jul 28, 2014 at 10:36 AM, Ruchir Jha <ruchir....@gmail.com> wrote: >> Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data >> Size = 21 TB. >> >> And this is the nodetool cfstats output for one of our busiest column >> families: >> >> SSTable count: 10 >> Space used (live): 43239294899 >> Space used (total): 43239419603 >> SSTable Compression Ratio: 0.2954468408497778 >> Number of Keys (estimate): 63729152 >> Memtable Columns Count: 1921620 >> Memtable Data Size: 257680020 >> Memtable Switch Count: 9 >> Read Count: 6167 >> Read Latency: NaN ms. >> Write Count: 770984 >> Write Latency: 0.098 ms. >> Pending Tasks: 0 >> Bloom Filter False Positives: 370 >> Bloom Filter False Ratio: 0.00000 >> Bloom Filter Space Used: 80103200 >> Compacted row minimum size: 180 >> Compacted row maximum size: 3311 >> Compacted row mean size: 2631 >> Average live cells per slice (last five minutes): 73.0 >> Average tombstones per slice (last five minutes): 13.0 >> >> >> >> On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <mark.re...@boxever.com> wrote: >>> What is your data size and number of columns in Cassandra. Do you do many >>> deletions? >>> >>> >>> On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ruchir....@gmail.com> wrote: >>>> Really curious to know what's causing the spike in Columns and >>>> DeletedColums below : >>>> >>>> >>>> 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class >>>> Histogram: >>>> num #instances #bytes class name >>>> ---------------------------------------------- >>>> 1: 132626060 6366050880 java.nio.HeapByteBuffer >>>> 2: 28194918 3920045528 [B >>>> 3: 78124737 3749987376 >>>> edu.stanford.ppl.concurrent.SnapTreeMap$Node >>>> 4: 67650128 2164804096 org.apache.cassandra.db.Column >>>> 5: 16315310 522089920 org.apache.cassandra.db.DeletedColumn >>>> 6: 6818 392489608 [I >>>> 7: 2844374 273059904 >>>> edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch >>>> 8: 5727000 229080000 java.util.TreeMap$Entry >>>> 9: 767742 182921376 [J >>>> 10: 2932832 140775936 >>>> edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder >>>> 11: 2844375 91020000 >>>> edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch >>>> 12: 4145131 66322096 >>>> java.util.concurrent.atomic.AtomicReference >>>> 13: 437874 64072392 [C >>>> 14: 2660844 63860256 >>>> java.util.concurrent.ConcurrentSkipListMap$Node >>>> 15: 4920 62849864 [[B >>>> 16: 1632063 52226016 edu.stanford.ppl.concurrent.SnapTreeMap >>> >> >
smime.p7s
Description: S/MIME cryptographic signature
