Number of compactions can spike for perfectly reasonable scenarios (ie after a repair or bootstrap) so no matter what you set it to there will likely be periodic false alarms.
I would however put a number "15" on the limit of any thread pool backlog as matter of principle until woken up too many times in middle of night. concurrent compactors will likely be to low (depending on number of cores). --- Chris Lohfink On Jul 14, 2014, at 7:31 PM, Greg Bone <gbon...@gmail.com> wrote: > I'm looking into creation of monitoring thresholds for cassandra to report on > its health. Does it make sense to set an alert threshold on compaction stats? > If so, would setting it to a value equal to or greater than concurrent > compactions make sense? > > Thanks, > Greg > > > > > On Mon, Jun 9, 2014 at 2:14 PM, S C <as...@outlook.com> wrote: > Thank you all for quick responses. > From: clohf...@blackbirdit.com > Subject: Re: high pending compactions > Date: Mon, 9 Jun 2014 14:11:36 -0500 > To: user@cassandra.apache.org > > Bean: org.apache.cassandra.db.CompactionManager > > also nodetool compactionstats gives you how many are in the queue + estimate > of how many will be needed. > > in 1.1 you will OOM far before you hit the limit,. In theory though, the > compaction executor is a little special cased and will actually throw an > exception (normally it will block) > > Chris > > On Jun 9, 2014, at 7:49 AM, S C <as...@outlook.com> wrote: > > Thank you all for valuable suggestions. Couple more questions, > > How to check the compaction queue? MBean/C* system log ? > What happens if the queue is full? > > From: colinkuo...@gmail.com > Date: Mon, 9 Jun 2014 18:53:41 +0800 > Subject: Re: high pending compactions > To: user@cassandra.apache.org > > As Jake suggested, you could firstly increase > "compaction_throughput_mb_per_sec" and "concurrent_compactions" to suitable > values if system resource is allowed. From my understanding, major compaction > will internally acquire lock before running compaction. In your case, there > might be a major compaction blocking the pending following compaction tasks. > You could check the result of "nodetool compactionstats" and C* system log > for double confirm. > > If the running compaction is compacting wide row for a long time, you could > try to tune "in_memory_compaction_limit_in_mb" value. > > Thanks, > > > > On Sun, Jun 8, 2014 at 11:27 PM, S C <as...@outlook.com> wrote: > I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending > compaction count. "pending tasks: 67" while active compaction tasks are not > more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? > Is this a pattern of high writes and compactions backing up? How can I > improve this? Here are my thoughts. > > Increase memtable_total_space_in_mb > Increase compaction_throughput_mb_per_sec > Increase concurrent_compactions > > Sorry if this was discussed already. Any pointers is much appreciated. > > Thanks, > Kumar > >
