Setting up a Flink metrics dashboard in Grafana requires setting up and configuring one of Flink's metrics reporters [1] that is supported by Grafana as a data source. That means your options for a metrics reporter are Graphite, InfluxDB, Prometheus, or the Prometheus push reporter.
If you want reporting every 5 seconds, with the push based reporters that's something you would configure in flink-conf.yaml, whereas with Prometheus you'll need to configure the scrape interval in the prometheus config file. For more on using Flink with Prometheus, see the blog post by Maximilian Bode [2]. Best, David [1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#reporter [2] https://flink.apache.org/features/2019/03/11/prometheus-monitoring.html On Fri, Jul 24, 2020 at 12:57 AM Vijay Balakrishnan <bvija...@gmail.com> wrote: > Hi, > I am trying to figure out how many records came into the Flink App from > KDS and how many records got moved to the next step or was dropped by the > watermarks. > > I see on the Ui Table for *Source. Records Sent* with a total and the > next step *Filter->FlatMap operator with a Records Received *total. How > can I get these metric values for me to display In Grafana for eg. as I > want to know a count for each 5 secs, how many records came in and how many > were filtered out by the watermark or my Custom Filter operator etc ? > > I looked at the breakdown of the Source__Custom_Source in Metrics as show > in the attached pic. It has values like 0.NumRecordsIn and 0.NumRecordsOut > and so on from 0 to 9 for the parallelism 10 I specified. It also has > various breakdowns like 0.Timestamps/Watermarks.numRecordsIn and > 0.Timestamps/Watermarks.numRecordsOut > > Attached are some screenshots of the Flink DashBoard UI. > > TIA, > >
