Hi Benjamin, This is somewhat of a hot topic lately, visibility FTW :)
My experience comes from doing Beam over Spark over Yarn, where we have been using Graphite + Grafana (4.0+ gives alerting as well). In terms of metrics, the Spark runner supports both Spark native metrics (default + Spark metric sources you configure, e.g. JVM), and Beam's, user-defined metrics, all of which are reported to whatever Spark sink(s) you have configured, in our case Graphite. There were a few caveats we had to address, most of which had to do with associating an application's logical name and its application id assigned by Yarn. In other words, making all the metrics reported by a given application id, go under its logical name required some effort. However, once it's been done it allows for dashboards to be built around applications' logical names rather than around specific application ids which is very useful. There are more caveats to speak of in the context of metrics and Beam over Spark (and streaming in particular), so let me know if you find the gory details interesting. -Stas On Thu, Mar 2, 2017 at 5:37 PM Amit Sela <[email protected]> wrote: > +Stas Levin <[email protected]> > > On Thu, Mar 2, 2017 at 5:30 PM Jean-Baptiste Onofré <[email protected]> > wrote: > > Hi Benjamin, > > It's a bit related to the Metric discussion on the dev@ mailing list. > > Today, we leverage the monitoring and management provided by the > execution engine of the runner. > > For instance, with the Spark runner, we can use the "regular" history > server, codehale reporter, etc. It's also possible to explicitly use a > sink/fn to send data to a backend (Graphite, elasticsearch, ...). > > We are now discussing about a agnostic way of "pushing" metric to a > backend (something similar to what Karaf Decanter provides). > > Regards > JB > > On 03/02/2017 04:13 PM, Stadin, Benjamin wrote: > > Hi, > > > > > > > > I’m trying to collect a list of open source monitoring and management > > tools for Beam and supported runners. > > > > > > > > What do you use in your daily routine, and what’s your experience? > > > > > > > > Regards > > > > Ben > > > > > > > > > > > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com > >
