+user for additional reach ---------- Forwarded message --------- From: Rion Williams <rionmons...@gmail.com> Date: Mon, Feb 15, 2021 at 7:58 PM Subject: Defining Custom Labels / Label Support in Beam Metrics To: dev <d...@beam.apache.org>
Hey all, I've been working extensively on the observability stories surrounding some of the pipelines that I've been building recently in Beam and while the existing Metrics have been extremely helpful, I don't see any easy or perhaps just well documented way of adding/updating labels for these metrics. What I was hoping to accomplish would be to just use these metrics and scape them into Prometheus/Grafana to create some nice dashboards, however in some of these use-cases, I need to be able to slice the data by certain attributes (e.g. a tenant, source, etc.) in addition to just overall/aggregate data. Does Beam have support for this? If not, what have folks been using to accomplish this? My first thought was to consider using the existing Prometheus Client Metrics [0], however if Beam had support for this, I'd much prefer that. Additionally, I'm not entirely sure what challenges might come with implementing metrics at the DoFn/Transform level using a third-party library such as this, given the distributed nature of Beam. Any insight / recommendations / thoughts would be greatly appreciated! Thanks, Rion [0] https://prometheus.io/docs/instrumenting/clientlibs/
