After some searching, I think I may have found something in the code itself, and so I've filed a big report - https://issues.apache.org/jira/browse/CASSANDRA-3711
Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com From: Caleb Rackliffe <ca...@steelhouse.com<mailto:ca...@steelhouse.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Sun, 8 Jan 2012 17:48:59 -0500 To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Cc: "aa...@thelastpickle.com<mailto:aa...@thelastpickle.com>" <aa...@thelastpickle.com<mailto:aa...@thelastpickle.com>> Subject: Re: Lots and Lots of CompactionReducer Threads With the exception of a few little warnings on start-up about the Memtable live ratio, there is nothing at WARN or above in the logs. Just before the JVM terminates, there are about 10,000 threads in Reducer executor pools that look like this in JConsole … Name: CompactionReducer:1 State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea Total blocked: 0 Total waited: 1 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) The results from tpstats don't look too interesting… Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 3455159 0 0 RequestResponseStage 0 0 10133276 0 0 MutationStage 0 0 5898833 0 0 ReadRepairStage 0 0 2078449 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 236388 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemtablePostFlusher 0 0 231 0 0 StreamStage 0 0 0 0 0 FlushWriter 0 0 231 0 0 MiscStage 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 35 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 REQUEST_RESPONSE 0 The results from info seem unremarkable as well… Token : 153127065000000000000000000000000000000 Gossip active : true Load : 5.6 GB Generation No : 1325995515 Uptime (seconds) : 67199 Heap Memory (MB) : 970.32 / 1968.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default on the number of concurrent compactors. What is interesting to me here is that Cassandra creates an executor for every single compaction in ParallelCompactionIterable. Why couldn't we just create a pool with Runtime.availableProcessors() Threads and be done with it? Let me know if I left any info out. Thanks! Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com<mailto:ca...@steelhouse.com> From: aaron morton <aa...@thelastpickle.com<mailto:aa...@thelastpickle.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Sun, 8 Jan 2012 16:51:50 -0500 To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Lots and Lots of CompactionReducer Threads How many threads ? Any errors in the server logs ? What does noodtool tpstats and nodetool compactionstats say ? Did you change compaction_strategy for the CF's ? By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml Have you set the JVM heap settings ? What does nodetool info show ? Hope that helps. ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote: Hi Everybody, JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup… Cassandra 1.0.5 on CentOS 6.0 4 GB of RAM 50 GB SSD HD Memtable flush threshold = 128 MB compaction throughput limit = 16 MB/sec Multithreaded compaction = true It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around because they're actually not done doing their compaction tasks. Thanks, Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com<mailto:ca...@steelhouse.com>
