Thanks, good work. Cheers
----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 9/01/2012, at 1:25 PM, Caleb Rackliffe wrote: > After some searching, I think I may have found something in the code itself, > and so I've filed a big report - > https://issues.apache.org/jira/browse/CASSANDRA-3711 > > Caleb Rackliffe | Software Developer > M 949.981.0159 | ca...@steelhouse.com > > > From: Caleb Rackliffe <ca...@steelhouse.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Sun, 8 Jan 2012 17:48:59 -0500 > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Cc: "aa...@thelastpickle.com" <aa...@thelastpickle.com> > Subject: Re: Lots and Lots of CompactionReducer Threads > > With the exception of a few little warnings on start-up about the Memtable > live ratio, there is nothing at WARN or above in the logs. Just before the > JVM terminates, there are about 10,000 threads in Reducer executor pools that > look like this in JConsole … > > > Name: CompactionReducer:1 > State: TIMED_WAITING on > java.util.concurrent.SynchronousQueue$TransferStack@72938aea > Total blocked: 0 Total waited: 1 > > Stack trace: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > java.lang.Thread.run(Thread.java:722) > > > The results from tpstats don't look too interesting… > > Pool Name Active Pending Completed Blocked All > time blocked > ReadStage 0 0 3455159 0 > 0 > RequestResponseStage 0 0 10133276 0 > 0 > MutationStage 0 0 5898833 0 > 0 > ReadRepairStage 0 0 2078449 0 > 0 > ReplicateOnWriteStage 0 0 0 0 > 0 > GossipStage 0 0 236388 0 > 0 > AntiEntropyStage 0 0 0 0 > 0 > MigrationStage 0 0 0 0 > 0 > MemtablePostFlusher 0 0 231 0 > 0 > StreamStage 0 0 0 0 > 0 > FlushWriter 0 0 231 0 > 0 > MiscStage 0 0 0 0 > 0 > InternalResponseStage 0 0 0 0 > 0 > HintedHandoff 0 0 35 0 > 0 > > Message type Dropped > RANGE_SLICE 0 > READ_REPAIR 0 > BINARY 0 > READ 0 > MUTATION 0 > REQUEST_RESPONSE 0 > > The results from info seem unremarkable as well… > > Token : 153127065000000000000000000000000000000 > Gossip active : true > Load : 5.6 GB > Generation No : 1325995515 > Uptime (seconds) : 67199 > Heap Memory (MB) : 970.32 / 1968.00 > Data Center : datacenter1 > Rack : rack1 > Exceptions : 0 > > I'm using LeveledCompactionStrategy with no throttling, and I'm not changing > the default on the number of concurrent compactors. > > What is interesting to me here is that Cassandra creates an executor for > every single compaction in ParallelCompactionIterable. Why couldn't we just > create a pool with Runtime.availableProcessors() Threads and be done with it? > > Let me know if I left any info out. > > Thanks! > > Caleb Rackliffe | Software Developer > M 949.981.0159 | ca...@steelhouse.com > > > From: aaron morton <aa...@thelastpickle.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Sun, 8 Jan 2012 16:51:50 -0500 > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: Lots and Lots of CompactionReducer Threads > > How many threads ? Any errors in the server logs ? > > What does noodtool tpstats and nodetool compactionstats say ? > > Did you change compaction_strategy for the CF's ? > > By default cassandra will use as many compaction threads as you have cores, > see concurrent_compactors in cassandra.yaml > > Have you set the JVM heap settings ? What does nodetool info show ? > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote: > >> Hi Everybody, >> >> JConsole tells me I've got CompactionReducer threads stacking up, consuming >> memory, and never going away. Eventually, my Java process fails because it >> can't allocate any more native threads. Here's my setup… >> >> Cassandra 1.0.5 on CentOS 6.0 >> 4 GB of RAM >> 50 GB SSD HD >> Memtable flush threshold = 128 MB >> compaction throughput limit = 16 MB/sec >> Multithreaded compaction = true >> >> It may very well be that I'm doing something strange here, but it seems like >> those compaction threads should go away eventually. I'm hoping the >> combination of a low Memtable flush threshold, low compaction T/P limit, and >> heavy write load doesn't mean those threads are hanging around because >> they're actually not done doing their compaction tasks. >> >> Thanks, >> >> Caleb Rackliffe | Software Developer >> M 949.981.0159 | ca...@steelhouse.com >> >
