Perhaps the deletes: https://issues.apache.org/jira/browse/CASSANDRA-3741
-Brandon On Sun, Jun 3, 2012 at 6:12 PM, Poziombka, Wade L <wade.l.poziom...@intel.com> wrote: > Running a very write intensive (new column, delete old column etc.) process > and failing on memory. Log file attached. > > Curiously when I add new data I have never seen this have in past sent > hundreds of millions "new" transactions. It seems to be when I modify. my > process is as follows > > key slice to get columns to modify in batches of 100, in separate threads > modify those columns. I advance the slice with the start key each with last > key in previous batch. Mutations done are update a column value in one > column family(token), delete column and add new column in another (pan). > > Runs well until after about 5 million rows then it seems to run out of > memory. Note that these column families are quite small. > > WARN [ScheduledTasks:1] 2012-06-03 17:49:01,558 GCInspector.java (line 145) > Heap is 0.7967470834946492 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java (line > 2772) Unable to reduce heap usage since there are no dirty column families > INFO [GossipStage:1] 2012-06-03 17:49:01,999 Gossiper.java (line 797) > InetAddress /10.230.34.170 is now UP > INFO [ScheduledTasks:1] 2012-06-03 17:49:10,048 GCInspector.java (line 122) > GC for ParNew: 206 ms for 1 collections, 7345969520 used; max is 8506048512 > INFO [ScheduledTasks:1] 2012-06-03 17:49:53,187 GCInspector.java (line 122) > GC for ConcurrentMarkSweep: 12770 ms for 1 collections, 5714800208 used; max > is 8506048512 > > ---------------- > Keyspace: keyspace > Read Count: 50042632 > Read Latency: 0.23157864418482224 ms. > Write Count: 44948323 > Write Latency: 0.019460829472992797 ms. > Pending Tasks: 0 > Column Family: pan > SSTable count: 5 > Space used (live): 1977467326 > Space used (total): 1977467326 > Number of Keys (estimate): 16334848 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 74 > Read Count: 14985122 > Read Latency: 0.408 ms. > Write Count: 19972441 > Write Latency: 0.022 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 829 > Bloom Filter False Ratio: 0.00073 > Bloom Filter Space Used: 37048400 > Compacted row minimum size: 125 > Compacted row maximum size: 149 > Compacted row mean size: 149 > > Column Family: token > SSTable count: 4 > Space used (live): 1250973873 > Space used (total): 1250973873 > Number of Keys (estimate): 14217216 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 49 > Read Count: 30059563 > Read Latency: 0.167 ms. > Write Count: 14985488 > Write Latency: 0.014 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 13642 > Bloom Filter False Ratio: 0.00322 > Bloom Filter Space Used: 28002984 > Compacted row minimum size: 150 > Compacted row maximum size: 258 > Compacted row mean size: 224 > > Column Family: counters > SSTable count: 2 > Space used (live): 561549994 > Space used (total): 561549994 > Number of Keys (estimate): 9985024 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 38 > Read Count: 4997947 > Read Latency: 0.092 ms. > Write Count: 9990394 > Write Latency: 0.023 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 191 > Bloom Filter False Ratio: 0.37525 > Bloom Filter Space Used: 18741152 > Compacted row minimum size: 125 > Compacted row maximum size: 179 > Compacted row mean size: 150 > > ----------------
