IMO there's a good case for both external monitoring tools and per-host minimalistic interface but I see Eric's point that every piece of code will require its maintenance. A cluster monitoring tool is definitely required. An embedded one has two nice properties: 1. It works out of the box 2. If I want to know NOW what's going on on the host, I don't have to poll like crazy (or wait 5 minutes for the next sample), I can just open the browser and see.
I'll see if I can get something simple running and ping back, but I can't commit on a timeline. Is adding jetty acceptable? Any other preferences? On Tue, May 4, 2010 at 8:49 PM, Anthony Molinaro < antho...@alumni.caltech.edu> wrote: > And just to show you what the dashboards look like here's a couple > of screen shots > > Jconsole like page of jvm stats > http://herbie.ddv.com/~anthonym/mondemand-2.png > > Cassandra specific memtable stats > http://herbie.ddv.com/~anthonym/mondemand-1.png > > -Anthony > > On Tue, May 04, 2010 at 10:03:52AM -0700, Michael Lum wrote: > > On 5/4/2010 7:21 AM, Eric Evans wrote: > > >On Tue, 2010-05-04 at 08:41 +0300, Ran Tavory wrote: > > >>How about the following compromise: > > >>Add a simple web server to each node with only one simple servlet that > > >>simply spits out all JMX stats on one page. Not fancy, no graphs, > > >>simply the same values you can get from jconsole, but on a web page. > > >>To me it seems like a fair tradeoff b/w maintenance and easier out of > > >>the box management. Shooting up jconsole for each server is > > >>cumbersome, at least in the environment I work in (firewalls, high > > >>latency etc) so a web interface can be nice. > > > > > >It still seems superfluous to me, but I'd be open to something > > >fire-and-forget (i.e. wouldn't need updating each time something new was > > >added). > > > > This is how we monitor our Cassandra clusters. Each Cassandra node runs > > a process that polls the JMX stats and then fires off events to a set of > > configured management nodes using either UDP or multicast, depending on > > the network. New Cassandra nodes in the same cluster and datacenter > > have the same config (and are configured centrally anyways), and the > > management nodes automatically add new nodes based on the events they > > receive, so all the graphs, dashboards, monitors, and downstream tools > > pick all of this up without needing a change. This way we don't need to > > fire up jconsole for hundreds of nodes and can do other interesting > > cluster-wide aggregations. Also, we don't have to remember to setup > > monitoring when the cluster grows. > > > > All the tools used are open source, and I'd be happy to share more > > detail if there is interest. > > -- > ------------------------------------------------------------------------ > Anthony Molinaro <antho...@alumni.caltech.edu> >
