And just to show you what the dashboards look like here's a couple of screen shots
Jconsole like page of jvm stats http://herbie.ddv.com/~anthonym/mondemand-2.png Cassandra specific memtable stats http://herbie.ddv.com/~anthonym/mondemand-1.png -Anthony On Tue, May 04, 2010 at 10:03:52AM -0700, Michael Lum wrote: > On 5/4/2010 7:21 AM, Eric Evans wrote: > >On Tue, 2010-05-04 at 08:41 +0300, Ran Tavory wrote: > >>How about the following compromise: > >>Add a simple web server to each node with only one simple servlet that > >>simply spits out all JMX stats on one page. Not fancy, no graphs, > >>simply the same values you can get from jconsole, but on a web page. > >>To me it seems like a fair tradeoff b/w maintenance and easier out of > >>the box management. Shooting up jconsole for each server is > >>cumbersome, at least in the environment I work in (firewalls, high > >>latency etc) so a web interface can be nice. > > > >It still seems superfluous to me, but I'd be open to something > >fire-and-forget (i.e. wouldn't need updating each time something new was > >added). > > This is how we monitor our Cassandra clusters. Each Cassandra node runs > a process that polls the JMX stats and then fires off events to a set of > configured management nodes using either UDP or multicast, depending on > the network. New Cassandra nodes in the same cluster and datacenter > have the same config (and are configured centrally anyways), and the > management nodes automatically add new nodes based on the events they > receive, so all the graphs, dashboards, monitors, and downstream tools > pick all of this up without needing a change. This way we don't need to > fire up jconsole for hundreds of nodes and can do other interesting > cluster-wide aggregations. Also, we don't have to remember to setup > monitoring when the cluster grows. > > All the tools used are open source, and I'd be happy to share more > detail if there is interest. -- ------------------------------------------------------------------------ Anthony Molinaro <antho...@alumni.caltech.edu>
