My first guess would be the bloom filter and index sampling from lots-o-rows
Check the row count in cfstats Check the bloom filter size in cfstats. Background on memory requirements http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 20/02/2013, at 11:27 PM, Andras Szerdahelyi <andras.szerdahe...@ignitionone.com> wrote: > Hey list, > > Any ideas ( before I take a heap dump ) what might be consuming my 8GB JVM > heap at startup in Cassandra 1.1.6 besides > row cache : not persisted and is at 0 keys when this warning is produced > Memtables : no write traffic at startup, my app's column families are > durable_writes:false > Pending tasks : no pending tasks, except for 928 compactions ( not sure where > those are coming from ) > I drew these conclusions from the StatusLogger output below: > > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,198 GCInspector.java (line 122) > GC for ConcurrentMarkSweep: 14959 ms for 2 collections, 7017934560 used; max > is 8375238656 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,198 StatusLogger.java (line 57) > Pool Name Active Pending Blocked > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,199 StatusLogger.java (line 72) > ReadStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,200 StatusLogger.java (line 72) > RequestResponseStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,200 StatusLogger.java (line 72) > ReadRepairStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,200 StatusLogger.java (line 72) > MutationStage 0 -1 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line 72) > ReplicateOnWriteStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line 72) > GossipStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line 72) > AntiEntropyStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line 72) > MigrationStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,201 StatusLogger.java (line 72) > StreamStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line 72) > MemtablePostFlusher 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line 72) > FlushWriter 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line 72) > MiscStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,202 StatusLogger.java (line 72) > commitlog_archiver 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,203 StatusLogger.java (line 72) > InternalResponseStage 0 0 0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line 77) > CompactionManager 0 928 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line 89) > MessagingService n/a 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line 99) > Cache Type Size Capacity > KeysToSave Provider > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,212 StatusLogger.java (line 100) > KeyCache 25 25 > all > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 106) > RowCache 0 0 > all org.apache.cassandra.cache.SerializingCacheProvider > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 113) > ColumnFamily Memtable ops,data > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 116) > MYAPP_1.CF 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 116) > MYAPP_2.CF 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 116) > HiveMetaStore.MetaStore 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 116) > system.NodeIdInfo 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 116) > system.IndexInfo 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,213 StatusLogger.java (line 116) > system.LocationInfo 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > system.Versions 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > system.schema_keyspaces 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > system.Migrations 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > system.schema_columnfamilies 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > system.schema_columns 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > system.HintsColumnFamily 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > system.Schema 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > cfs_archive.sblocks 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,214 StatusLogger.java (line 116) > cfs_archive.cleanup 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > cfs_archive.inode 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > cfs.cleanup 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > cfs.sblocks 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > cfs.inode 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > OpsCenter.events 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > OpsCenter.rollups300 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > OpsCenter.rollups86400 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > OpsCenter.pdps 345,54448 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,215 StatusLogger.java (line 116) > OpsCenter.events_timeline 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line 116) > OpsCenter.settings 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line 116) > OpsCenter.rollups7200 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line 116) > OpsCenter.rollups60 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line 116) > demo.users 0,0 > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,216 StatusLogger.java (line 116) > MYAPP_3.CF 0,0 > WARN [ScheduledTasks:1] 2013-02-20 05:13:25,216 GCInspector.java (line 145) > Heap is 0.8379384574280004 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN [ScheduledTasks:1] 2013-02-20 05:13:25,217 StorageService.java (line > 2855) Flushing CFS(Keyspace='OpsCenter', ColumnFamily='pdps') to relieve > memory pressure > INFO [ScheduledTasks:1] 2013-02-20 05:13:25,217 ColumnFamilyStore.java (line > 659) Enqueuing flush of Memtable-pdps@98567689(15238/54448 serialized/live > bytes, 345 ops) > INFO [FlushWriter:6] 2013-02-20 05:13:25,218 Memtable.java (line 264) > Writing Memtable-pdps@98567689(15238/54448 serialized/live bytes, 345 ops) > INFO [FlushWriter:6] 2013-02-20 05:13:25,241 Memtable.java (line 305) > Completed flushing > /var/lib/cassandra/data/OpsCenter/pdps/OpsCenter-pdps-hf-127-Data.db (5394 > bytes) for commitlog position ReplayPosition(segmentId=1361354508149, : > > > > Thanks! > Andras >
