On Thu, May 20, 2010 at 02:11:23PM -0700, Jonathan Ellis wrote: > No, CM is not exposed to nodetool yet. (You should really be putting > metrics into a real monitoring system rather than relying on nodetool. > Some example munin plugins are at > http://github.com/jbellis/cassandra-munin-plugins, for instance.) > > CM also has BytesCompacted/BytesTotalInProgress.
I actually do put metrics into a monitoring system, but often use nodetool to see what's happening as well. However, the system I use to poll jmx is not finding BytesCompacted/BytesTotalInProgress (I mentioned the system we used in a previous email with some screenshots). So I downloaded jmxquery.jar and ran it, which gave me this. % java -jar ./jmxquery.jar \ --url=service:jmx:rmi:///jndi/rmi://localhost:32123/jmxrmi | \ grep -i compaction org.apache.cassandra.db:type=CompactionManager%PendingTasks.value 0 org.apache.cassandra.db:type=CompactionManager%MinimumCompactionThreshold.value 4 org.apache.cassandra.db:type=CompactionManager%MaximumCompactionThreshold.value 32 org.apache.cassandra.db:type=CompactionManager%ColumnFamilyInProgress.value null org.apache.cassandra.db:type=CompactionManager%BytesTotalInProgress.value null org.apache.cassandra.db:type=CompactionManager%BytesCompacted.value null Which explains why I never see it, our poller ignores 'null's So why are the Bytes* entries null? (I'm using 0.6.2 with the the first 4 bugs fixed 992, 1024 x 2, 1039). Is this something which only works with trunk? > Backup in deserialize is likely to be (a) the read or write stage > being full so deserialize in turn backs up. Anything that sucks up > enough CPU could also cause it, I suppose, but other than that > (generating lots of CPU load) there is nothing special about > compactions wrt deserialize. Is there some metric I could look at to determine if it's a? CPU is usually at most 20% on these boxes, so I assume it's a causing the backup. -Anthony > On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro > <antho...@alumni.caltech.edu> wrote: > > Hi, > > > > In the 0.5.x series there was a COMPACTION-POOL which kept track of > > in process, pending and completed compactions. With 0.6.x this seems > > to have vanished and instead we only have the CompactionManager PendingTasks > > statistic. Is there also a completed tasks somewhere? Is there any > > way to determine via nodetool if a compaction or anticompaction is occuring? > > I'm trying to figure out the cause of random spikes in the Message > > Deserializer > > Queue, and wanted to know if there was any correlation with compaction, > > but with only PendingTasks, I would have to be sampling continuously to > > catch it, as pending is mostly zero. > > > > Are there other possible causes for a backup in Message Deserializer? > > My traffic patterns are pretty constant, I don't see spikes in the > > thread pool completed tasks, and often the rate is below the high water > > mark for rates. > > > > Thanks, > > > > -Anthony > > > > -- > > ------------------------------------------------------------------------ > > Anthony Molinaro <antho...@alumni.caltech.edu> > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com -- ------------------------------------------------------------------------ Anthony Molinaro <antho...@alumni.caltech.edu>
