Jungtaek, I think filters that can support a regex gives more felxibility. Thanks, Harsha On Mon, May 2, 2016, at 07:48 PM, Jungtaek Lim wrote: > Kevin, > > For specific task, you can register your own metrics which resides > per task. > But metrics doc on Storm is not kind enough to let users follow, so I > addressed this and submitted pull request. > https://github.com/HeartSaVioR/storm/blob/STORM-1724-1.x/docs/Metrics.md > > There're no custom worker metrics for users since Storm abstracts > user's logic into 'task' so normally users don't need to measure JVM > level metrics (except JMX metrics). But it would be possible to add if > it's really great. > > Does current Storm work for you now? Or we would like to address / > improve something? > > Stephen and Harsha, > > Improve MetricsConsumer is an umbrella issue, and it contains the > feature 'blacklist & whitelist of metrics[1]'. > For now it filters metrics by only metrics name, but adding filter > targets to component name, host, etc. are easy so I would like to see > the needs before making the change. > Do you think adding filter targets makes sense? Or we can just start > with metrics name filter? > > Thanks, > Jungtaek Lim (HeartSaVioR) > > > 2016년 5월 3일 (화) 오전 2:18, Harsha <[email protected]>님이 작성: >> __ >> Jungtaek, >> Probably a filter config to whitelist and blacklist certain metrics. >> So that it will scale if there are too many workers and users can >> turn off certain metrics. >> >> Thanks, >> Harsha >> >> >> On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote: >>> Oooh I'd love this as well! I really dig the ease of the metric >>> framework in storm and have all the metrics go thru one centralized >>> config. But as the number of storm hosts and number of tasks grow, >>> I've found that Graphite/Grafana has a hard time collecting up all >>> the relevant metrics across a lot of wildcarded keys for things like >>> hostnames and taskIds to properly display my graphs. >>> >>> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway >>> <[email protected]> wrote: >>>> One thing I would like to see added (if not already present) is the >>>> ability to register metrics that are not tied to a component. >>>> >>>> As of now, the only non-component metrics are reported by the >>>> SystemBolt pseudo-component which feels like a work-around. It >>>> reports JVM level metrics like GC time, heap size and other things >>>> that aren't associated with a given component. >>>> >>>> It would be great if application developers could expose similar >>>> metrics like this for things like connection pools and other JVM >>>> wide objects that aren't unique to a specific component. >>>> >>>> I don't think this is possible now, is it? >>>> >>>> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <[email protected]> >>>> wrote: >>>>> Let me start sharing my thought. :) >>>>> >>>>> 1. Need to enrich docs about metrics / stats. >>>>> >>>>> In fact, I couldn't see the fact - topology stats are sampled by >>>>> default and sample rate is 0.05 - from the docs when I was newbie >>>>> of Apache Storm. It made me misleading and made me saying "Why >>>>> there're difference between the counts?". I also saw some mails >>>>> from user@ about same question. If we include this to guide doc >>>>> that would be better. >>>>> >>>>> And Metrics document page[2] seems not well written. I think it >>>>> has appropriate headings but lacks contents on each heading. >>>>> It should be addressed, and introducing some external metrics >>>>> consumer plugins (like storm-graphite[3] from Verisign) would be >>>>> great, too. >>>>> >>>>> 2. Need to increase sample rate or (ideally) no sampling at all. >>>>> >>>>> Let's postpone considering performance hit at this time. >>>>> Ideally, we expect precision of metrics gets better when we >>>>> increase sample rate. It affects non-gauge kinds of metrics which >>>>> are counter, and latency, and so on. >>>>> >>>>> Btw, I would like to hear about opinions on latency since I'm not >>>>> an expert. >>>>> Storm provides only average latency and it's indeed based on >>>>> sample rate. Do we feel OK with this? If not how much having also >>>>> percentiles can help us? >>>>> >>>>> Thanks, >>>>> Jungtaek Lim (HeartSaVioR) >>>>> >>>>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <[email protected]>님이 작성: >>>>>> Hi Storm users, >>>>>> >>>>>> I'm Jungtaek Lim, committer and PMC member of Apache Storm. >>>>>> >>>>>> If you subscribed dev@ mailing list, you may have seen that >>>>>> recently we're addressing the metrics feature on Apache Storm. >>>>>> >>>>>> For now, improvements are going forward based on current metrics >>>>>> feature. >>>>>> >>>>>> - Improve (Topology) MetricsConsumer[4] >>>>>> - Provide topology metrics in detail (metrics per each stream)[5] >>>>>> - (WIP) Introduce Cluster Metrics Consumer >>>>>> >>>>>> As I don't maintain large cluster for myself, I really want to >>>>>> collect the any ideas for improving, any inconveniences, use >>>>>> cases of Metrics with community members, so we're on the right >>>>>> way to go forward. >>>>>> >>>>>> Let's talk! >>>>>> >>>>>> Thanks in advance, >>>>>> Jungtaek Lim (HeartSaVioR) >>>> >>>> >>>> >>>> -- >>>> Kevin Conaway http://www.linkedin.com/pub/kevin-conaway/7/107/580/ >>>> https://github.com/kevinconaway >>>> >>
Links: 1. https://issues.apache.org/jira/browse/STORM-1700 2. http://storm.apache.org/releases/1.0.0/Metrics.html 3. https://github.com/verisign/storm-graphite 4. https://issues.apache.org/jira/browse/STORM-1699 5. https://issues.apache.org/jira/browse/STORM-1719
