> - still getting 1-2 sstable reads with LCS That's bang in the wheel house http://www.datastax.com/dev/blog/when-to-use-leveled-compaction
> However at startup I see a 5GB old gen ( that seems to be very stable at > around 5.5GB under moderate 90:10 read:write load - couple hundred q/s ) At 300 to 400 million rows that sounds roughly right. I've not done any calculations, but I just spilt a coffee and checked the stain it made. The interesting thing is what happens at the end of a CMS run. How low does it get ? > Also hinted handoff is off, as full GC pauses of 1-3seconds on nodes I have > not yet attended to I would look at the first. Search the user group for "correlate compaction gc" as a starting point. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 26/02/2013, at 3:22 AM, Andras Szerdahelyi <andras.szerdahe...@ignitionone.com> wrote: > ( I am monitoring via a visual vm plugin that shows generation sizes ) > I've increased the index sampling rate from 64 to 512 and the bloom filter fp > ratio from default to 0.1 ( and rebuilt stables ) - still getting 1-2 sstable > reads with LCS > However at startup I see a 5GB old gen ( that seems to be very stable at > around 5.5GB under moderate 90:10 read:write load - couple hundred q/s ) is > that a realistic size with these values and a ~200GB CF with 3-400mil rows? ( > no secondary indexes, skinny rows 1-5K each ) ? > > Also hinted handoff is off, as full GC pauses of 1-3seconds on nodes I have > not yet attended to ( sampling rate at 64, bloom fp default ) are kicking off > the failure detector even at a phi_convict_threshold of 12 ( apparently ? ) > and everything is in an up/down dance :-) > > Thanks! > Andras > > From: aaron morton <aa...@thelastpickle.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Friday 22 February 2013 17:44 > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: "Heap is N.N full." Immediately on startup > > To get a good idea of how GC is performing turn on the GC logging in > cassandra-env.sh. > > After a full cms GC event, see how big the tenured heap is. If it's not > reducing enough then GC will never get far enough ahead. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 22/02/2013, at 8:37 AM, Andras Szerdahelyi > <andras.szerdahe...@ignitionone.com> wrote: > >> Thank you- indeed my index interval is 64 with a CF of 300M rows + bloom >> filter false positive chance was default. >> Raising the index interval to 512 didn't fix this alone, so I guess I'll >> have to set the bloom filter to some reasonable value and scrub. >> >> From: aaron morton <aa...@thelastpickle.com> >> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Date: Thursday 21 February 2013 17:58 >> To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Subject: Re: "Heap is N.N full." Immediately on startup >> >> My first guess would be the bloom filter and index sampling from lots-o-rows >> >> Check the row count in cfstats >> Check the bloom filter size in cfstats. >> >> Background on memory requirements >> http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 20/02/2013, at 11:27 PM, Andras Szerdahelyi >> <andras.szerdahe...@ignitionone.com> wrote: >> >>> Hey list, >>> >>> Any ideas ( before I take a heap dump ) what might be consuming my 8GB JVM >>> heap at startup in Cassandra 1.1.6 besides >>> row cache : not persisted and is at 0 keys when this warning is produced >>> Memtables : no write traffic at startup, my app's column families are >>> durable_writes:false >>> Pending tasks : no pending tasks, except for 928 compactions ( not sure >>> where those are coming from ) >>> I drew these conclusions from the StatusLogger output below: >>> >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,198 GCInspector.java (line 122) >>> GC for ConcurrentMarkSweep: 14959 ms for 2 collections, 7017934560 used; >>> max is 8375238656 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,198 StatusLogger.java (line >>> 57) Pool Name Active Pending Blocked >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,199 StatusLogger.java (line >>> 72) ReadStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,200 StatusLogger.java (line >>> 72) RequestResponseStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,200 StatusLogger.java (line >>> 72) ReadRepairStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,200 StatusLogger.java (line >>> 72) MutationStage 0 -1 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line >>> 72) ReplicateOnWriteStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line >>> 72) GossipStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line >>> 72) AntiEntropyStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line >>> 72) MigrationStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line >>> 72) StreamStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line >>> 72) MemtablePostFlusher 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line >>> 72) FlushWriter 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line >>> 72) MiscStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line >>> 72) commitlog_archiver 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,203 StatusLogger.java (line >>> 72) InternalResponseStage 0 0 0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line >>> 77) CompactionManager 0 928 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line >>> 89) MessagingService n/a 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line >>> 99) Cache Type Size Capacity >>> KeysToSave >>> Provider >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line >>> 100) KeyCache 25 25 >>> all >>> >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 106) RowCache 0 0 >>> all >>> org.apache.cassandra.cache.SerializingCacheProvider >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 113) ColumnFamily Memtable ops,data >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 116) MYAPP_1.CF 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 116) MYAPP_2.CF 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 116) HiveMetaStore.MetaStore 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 116) system.NodeIdInfo 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 116) system.IndexInfo 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line >>> 116) system.LocationInfo 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) system.Versions 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) system.schema_keyspaces 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) system.Migrations 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) system.schema_columnfamilies 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) system.schema_columns 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) system.HintsColumnFamily 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) system.Schema 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) cfs_archive.sblocks 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line >>> 116) cfs_archive.cleanup 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) cfs_archive.inode 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) cfs.cleanup 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) cfs.sblocks 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) cfs.inode 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) OpsCenter.events 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) OpsCenter.rollups300 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) OpsCenter.rollups86400 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) OpsCenter.pdps 345,54448 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line >>> 116) OpsCenter.events_timeline 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line >>> 116) OpsCenter.settings 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line >>> 116) OpsCenter.rollups7200 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line >>> 116) OpsCenter.rollups60 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line >>> 116) demo.users 0,0 >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line >>> 116) MYAPP_3.CF 0,0 >>> WARN [ScheduledTasks:1] 2013-02-20 05:13:25,216 GCInspector.java (line >>> 145) Heap is 0.8379384574280004 full. You may need to reduce memtable >>> and/or cache sizes. Cassandra will now flush up to the two largest >>> memtables to free up memory. Adjust flush_largest_memtables_at threshold >>> in cassandra.yaml if you don't want Cassandra to do this automatically >>> WARN [ScheduledTasks:1] 2013-02-20 05:13:25,217 StorageService.java (line >>> 2855) Flushing CFS(Keyspace='OpsCenter', ColumnFamily='pdps') to relieve >>> memory pressure >>> INFO [ScheduledTasks:1] 2013-02-20 05:13:25,217 ColumnFamilyStore.java >>> (line 659) Enqueuing flush of Memtable-pdps@98567689(15238/54448 >>> serialized/live bytes, 345 ops) >>> INFO [FlushWriter:6] 2013-02-20 05:13:25,218 Memtable.java (line 264) >>> Writing Memtable-pdps@98567689(15238/54448 serialized/live bytes, 345 ops) >>> INFO [FlushWriter:6] 2013-02-20 05:13:25,241 Memtable.java (line 305) >>> Completed flushing >>> /var/lib/cassandra/data/OpsCenter/pdps/OpsCenter-pdps-hf-127-Data.db (5394 >>> bytes) for commitlog position ReplayPosition(segmentId=1361354508149, : >>> >>> >>> >>> Thanks! >>> Andras >>> >> >
