Hi, Robert, I think I found the cause of the too many compactions. I used jmap to dump the heap and used Eclipse memory analyzer plugin to extract the heap.
In previous reply, It shows the there are too many pending jobs in the Blocking queue. I checked the cf of the compaction task object. There are many cfs concerning some empty cfs I created before. I created 5 keyspaces and about 100 cfs months by cassandra-cli ago and didnot put any data yet. In fact, there is only 1 keypaces I created containing data and the other 5 keyspaces are empty. When I droped these 5 keyspaces and restarted the high compaction node, It runs normally with normal mount of compactions. So maybe there are some bugs of compaction for empty columnfamily? On Wed, Jan 13, 2016 at 2:39 AM, Robert Coli <rc...@eventbrite.com> wrote: > On Mon, Jan 11, 2016 at 9:12 PM, Shuo Chen <chenatu2...@gmail.com> wrote: > >> I have a assumption that, lots of pending compaction tasks jam the memory >> and raise full gc. The full chokes the process and slows down compaction. >> And this causes more pending compaction tasks and more pressure on memory. >> > > The question is why there are so many pending compactions, because your > log doesn't show that much compaction is happening. What keyspaces / > columnfamilies do you expect to be compacting, and how many SSTables do > they contain? > > >> Is there a method to list the concrete details of pending compaction >> tasks? >> > > Nope. > > For the record, this type of extended operational debugging is often best > carried out interactively on #cassandra on freenode IRC.. :) > > =Rob > -- *陈硕* *Shuo Chen* chenatu2...@gmail.com chens...@whaty.com
