aaron morton <aaron <at> thelastpickle.com> writes: > > > > I've been playing around with trying to figure out what is making compactions run so slow.Is this regular compaction or table upgrades ? > I *think* upgrade tables is single threaded. > Do you have some compaction logs lines that say "Compacted to…"? It's handy > to see the throughput and the number of keys compacted. > > snapshot_before_compaction: falsein_memory_compaction_limit_in_mb: 256multithreaded_compaction: truecompaction_throughput_mb_per_sec: 128compaction_preheat_key_cache: true > What setting for concurrent_compactors ? > I would also check the logs for GC issues. > > Cheers > > > > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > <at> aaronmorton > http://www.thelastpickle.com > > > > > On 28/11/2012, at 4:23 AM, Derek Bromenshenkel <derek.bromenshenkel <at> gmail.com> wrote: > Setup: C* 1.1.6, 6 node (Linux, 64GB RAM, 16 Core CPU, 2x512 SSD), RF=3, 1.65TB total usedBackground: Client app is off - no reads/writes happening. Doing some cluster maintenance requiring node repairs and upgradesstables.I've been playing around with trying to figure out what is making compactions run so slow. Watching syslogs, it seems to average 3-4MB/s. That just seems so slow for this set up and the fact there is zero external load on the cluster. As far as I can tell:1. Not I/O bound according to iostat data 2. CPU seems to be idiling also3. From my understanding, I am using all the correct compaction settings for this setup: Here are those below:snapshot_before_compaction: falsein_memory_compaction_limit_in_mb: 256multithreaded_compaction: truecompaction_throughput_mb_per_sec: 128compaction_preheat_key_cache: trueSome other thoughts:- I have turned on DEBUG logging for the Throttle class and played with the live compaction_throughput_mb_per_sec setting. I can see it performing the throttling if I set the value low (say 4), but anything over 8 it is apparently running wide open. [Side note: Although the math for the Throttle class adds up, over all the throttling seems to be very very conservative.]- I accidently turned on DEBUG for the entire ...compaction.* package and that unintentionally created A LOT of I/O from the ParallelCompactionIterable class, and the disk/OS handled that just fine.Perhaps I just don't fully grasp what is going on or have the correct expectations. I am OK with things being slow if the hardware is working hard, but that does not seem to be the case.Anyone have some insight?Thanks > > > > > >
Hi Aaron, Thank you for taking the time and responding. I'll try to answer your questions. - reg vs upgrade: Seeing the same speed on regular compaction and upgrades. True that most of the frustration comes from the upgrades since there is so much work to be done. - GC: looked fine. I've seen pressure before, but only when under very heavy client app load. - concurrent_compactors: not set, so should default to #cores [32; 16 phys * 2 hyperthread], and I did see 32 CompactionExecuter (I think) threads via JMX. - examples: yes I have a lot of examples. here are some Leveled INFO [CompactionExecutor:1033] 2012-11-26 01:38:56,800 CompactionTask.java (line 221) Compacted to [lcs1]. 35,058,450 to 33,408,896 (~95% of original) bytes for 127,771 keys at 4.371735MB/s. Time: 7,288ms. INFO [CompactionExecutor:2015] 2012-11-26 03:12:43,800 CompactionTask.java (line 221) Compacted to [lcs2]. 37,029,581 to 36,747,459 (~99% of original) bytes for 135,471 keys at 3.748541MB/s. Time: 9,349ms. Size Tiered INFO [CompactionExecutor:6242] 2012-11-26 10:46:24,030 CompactionTask.java (line 221) Compacted to [abc]. 12,804,781,130 to 5,575,340,207 (~43% of original) bytes for 84,723,404 keys at 1.382544MB/s. Time: 3,845,851ms. INFO [CompactionExecutor:288] 2012-11-26 00:42:58,629 CompactionTask.java (line 221) Compacted to [def]. 116,347,764 to 58,354,237 (~50% of original) bytes for 2,511,375 keys at 0.655612MB/s. Time: 84,884ms. INFO [CompactionExecutor:5113] 2012-11-26 08:33:12,885 CompactionTask.java (line 221) Compacted to [ghi]. 560,682,371 to 294,965,985 (~52% of original) bytes for 220 keys at 3.172669MB/s. Time: 88,664ms. INFO [CompactionExecutor:6124] 2012-11-26 09:36:52,141 CompactionTask.java (line 221) Compacted to [jkl]. 418,807,103 to 234,394,618 (~55% of original) bytes for 3,130,751 keys at 2.807220MB/s. Time: 79,629ms. Also, upon reading the messages here/JIRA/etc I decided to disable multithreaded_compaction late yesterday. That helped to the tune of 3-5x improvement. Why multi is so much slower I'm willing to digress for now. However, I'm still interested in understanding why, under zero load in an unthrottled state, the compaction process does not consume at least one full CPU core and/or max out the disk I/O. Thanks again, Derek
