Peter Schuller <peter.schuller <at> infidyne.com> writes: > Without necessarily dumping all the information - approximately what > do they contain? Do they contain anything about compactions, > anti-compactions, streaming, etc? > > With an idle node after taking writes, I *think* the only expected > disk I/O (once it has settled) would be a memtable flush triggered by > memtable_flush_after_mins, and possibly compactions resulting from > that (depending on how close one were to triggering compaction prior > to the memtable flush). Whatever is causing additional sstables to be > written, even if somehow triggered incorrectly, I'd hope that they > were logged still.
****************************************************** Hi Peter, Thanks for your ideas. Here is what I see in the bloated node's system.log file (condensed) starting 4 minutes after the last write. At this time, data usage on this node was only around 28GB (as expected). ****************************************************** INFO [COMPACTION-POOL:1] 2010-08-16 13:51:38,432 CompactionManager.java (line 246) Compacting ... INFO [COMPACTION-POOL:1] 2010-08-16 14:20:02,840 CompactionManager.java (line 246) Compacting ... INFO [COMPACTION-POOL:1] 2010-08-16 14:28:37,085 CompactionManager.java (line 246) Compacting ... INFO [Thread-25] 2010-08-16 14:34:06,827 StreamCompletionHandler.java (line 64) Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-467-Data.db INFO [Timer-1] 2010-08-16 14:34:31,655 Gossiper.java (line 179) InetAddress /10.254.107.178 is now dead. INFO [GMFD:1] 2010-08-16 14:35:03,012 Gossiper.java (line 568) InetAddress /10.254.107.178 is now UP INFO [COMPACTION-POOL:1] 2010-08-16 14:36:04,808 CompactionManager.java (line 246) Compacting ... INFO [COMPACTION-POOL:1] 2010-08-16 14:40:31,283 CompactionManager.java (line 246) Compacting ... INFO [FLUSH-TIMER] 2010-08-16 14:50:46,570 ColumnFamilyStore.java (line 357) Standard1 has reached its threshold; switching in a fresh Memtable at CommitLogContext(file= '/var/lib/cassandra/commitlog/CommitLog-1281980828674.log', position=87493038) INFO [FLUSH-TIMER] 2010-08-16 14:50:46,602 ColumnFamilyStore.java (line 609) Enqueuing flush of Memtable(Standard1)@31873581 INFO [FLUSH-WRITER-POOL:1] 2010-08-16 14:50:46,647 Memtable.java (line 148) Writing Memtable(Standard1)@31873581 INFO [GC inspection] 2010-08-16 14:50:47,350 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 362 ms, 506927464 reclaimed leaving 129128920 used; max is 1172766720 INFO [FLUSH-WRITER-POOL:1] 2010-08-16 14:50:51,137 Memtable.java (line 162) Completed flushing /var/lib/cassandra/data/Keyspace1/Standard1-472-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:50:58,411 SSTableDeletingReference.java (line 104) Deleted /var/lib/cassandra/data/Keyspace1/Standard1-223-Data.db ... ... INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:51:40,662 SSTableDeletingReference.java (line 104) Deleted /var/lib/cassandra/data/Keyspace1/Standard1-407-Data.db INFO [Timer-1] 2010-08-16 14:51:40,772 Gossiper.java (line 179) InetAddress /10.254.242.159 is now dead. INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:51:41,011 SSTableDeletingReference.java (line 104) Deleted /var/lib/cassandra/data/Keyspace1/Standard1-279-Data.db ... ... INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:52:22,206 SSTableDeletingReference.java (line 104) Deleted /var/lib/cassandra/data/Keyspace1/Standard1-389-Data.db INFO [GMFD:1] 2010-08-16 14:52:34,906 Gossiper.java (line 568) InetAddress /10.254.242.159 is now UP INFO [COMPACTION-POOL:1] 2010-08-16 14:58:48,958 CompactionManager.java (line 246) Compacting ... INFO [COMPACTION-POOL:1] 2010-08-16 15:02:43,444 CompactionManager.java (line 246) Compacting ... INFO [COMPACTION-POOL:1] 2010-08-16 15:05:08,837 CompactionManager.java (line 246) Compacting [] INFO [Timer-1] 2010-08-16 15:17:38,697 Gossiper.java (line 179) InetAddress /10.254.234.226 is now dead. INFO [GMFD:1] 2010-08-16 15:18:37,471 Gossiper.java (line 568) InetAddress /10.254.234.226 is now UP INFO [Thread-28] 2010-08-16 15:38:34,685 StreamCompletionHandler.java (line 64) Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-475-Data.db INFO [COMPACTION-POOL:1] 2010-08-16 16:05:08,811 CompactionManager.java (line 246) Compacting [] INFO [COMPACTION-POOL:1] 2010-08-16 17:05:08,792 CompactionManager.java (line 246) Compacting [] INFO [Timer-1] 2010-08-16 17:05:55,583 Gossiper.java (line 179) InetAddress /10.254.242.159 is now dead. INFO [GMFD:1] 2010-08-16 17:06:19,614 Gossiper.java (line 568) InetAddress /10.254.242.159 is now UP INFO [Thread-34] 2010-08-16 17:24:57,584 StreamCompletionHandler.java (line 64) Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-477-Data.db INFO [Timer-1] 2010-08-16 17:25:19,206 Gossiper.java (line 179) InetAddress /10.210.157.187 is now dead. INFO [GMFD:1] 2010-08-16 17:25:53,096 Gossiper.java (line 568) InetAddress /10.210.157.187 is now UP INFO [Thread-31] 2010-08-16 17:26:37,281 StreamCompletionHandler.java (line 64) Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-476-Data.db INFO [Timer-1] 2010-08-16 17:27:01,939 Gossiper.java (line 179) InetAddress /10.254.234.226 is now dead. INFO [GMFD:1] 2010-08-16 17:27:24,115 Gossiper.java (line 568) InetAddress /10.254.234.226 is now UP INFO [COMPACTION-POOL:1] 2010-08-16 18:05:08,764 CompactionManager.java (line 246) Compacting [] INFO [COMPACTION-POOL:1] 2010-08-16 19:05:08,741 CompactionManager.java (line 246) Compacting [] INFO [COMPACTION-POOL:1] 2010-08-16 20:05:08,718 CompactionManager.java (line 246) Compacting [] INFO [COMPACTION-POOL:1] 2010-08-16 21:05:08,694 CompactionManager.java (line 246) Compacting [] INFO [COMPACTION-POOL:1] 2010-08-16 22:05:08,674 CompactionManager.java (line 246) Compacting [] ****************************************************** At this point, I logged in. The data distribution on this node was 122GB. I started performing a manual nodetool cleanup. ******************************************************
