> I've been playing around with trying to figure out what is making compactions > run so slow. Is this regular compaction or table upgrades ? I *think* upgrade tables is single threaded.
Do you have some compaction logs lines that say "Compacted to…"? It's handy to see the throughput and the number of keys compacted. > snapshot_before_compaction: false > in_memory_compaction_limit_in_mb: 256 > multithreaded_compaction: true > compaction_throughput_mb_per_sec: 128 > compaction_preheat_key_cache: true What setting for concurrent_compactors ? I would also check the logs for GC issues. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 28/11/2012, at 4:23 AM, Derek Bromenshenkel <derek.bromenshen...@gmail.com> wrote: > Setup: C* 1.1.6, 6 node (Linux, 64GB RAM, 16 Core CPU, 2x512 SSD), RF=3, > 1.65TB > total used > Background: Client app is off - no reads/writes happening. Doing some cluster > maintenance requiring node repairs and upgradesstables. > > I've been playing around with trying to figure out what is making compactions > run so slow. Watching syslogs, it seems to average 3-4MB/s. That just seems > so > slow for this set up and the fact there is zero external load on the cluster. > > As far as I can tell: > 1. Not I/O bound according to iostat data > 2. CPU seems to be idiling also > 3. From my understanding, I am using all the correct compaction settings for > this setup: Here are those below: > > snapshot_before_compaction: false > in_memory_compaction_limit_in_mb: 256 > multithreaded_compaction: true > compaction_throughput_mb_per_sec: 128 > compaction_preheat_key_cache: true > > Some other thoughts: > - I have turned on DEBUG logging for the Throttle class and played with the > live > compaction_throughput_mb_per_sec setting. I can see it performing the > throttling if I set the value low (say 4), but anything over 8 it is > apparently > running wide open. [Side note: Although the math for the Throttle class adds > up, > over all the throttling seems to be very very conservative.] > - I accidently turned on DEBUG for the entire ...compaction.* package and > that > unintentionally created A LOT of I/O from the ParallelCompactionIterable > class, > and the disk/OS handled that just fine. > > Perhaps I just don't fully grasp what is going on or have the correct > expectations. I am OK with things being slow if the hardware is working > hard, > but that does not seem to be the case. > > Anyone have some insight? > > Thanks >
