Thanks for all your replies.Ended up using a StatsdReporter with Flink and building a statsd plugin to transform the data to my required output format and dump it into a folder that the Kinesis agent can then pick up.
On Tue, Jul 16, 2019 at 2:16 AM Chesnay Schepler <ches...@apache.org> wrote: > You can configure multiple reporters, so just configured both the reporter > that the app users want and your own which does the transformation and > sending to kinesis. > > On 16/07/2019 09:37, Haibo Sun wrote: > > Hi, Vijay > > Or can you implement a Reporter that transforms the metrics and sends them > directly to a Kinesis Stream? > > Best, > Haibo > > At 2019-07-16 00:01:36, "Vijay Balakrishnan" <bvija...@gmail.com> > <bvija...@gmail.com> wrote: > > Hi, > I need to capture the Metrics sent from a Flink app to a Reporter and > transform them to an Events API format I have designed. I have been looking > at the Reporters( > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables) > and have used them but what would be a best practice to capture this > metrics data to transform it ? > > The folks using the Flink app still want to see their metrics in the Flink > Dashboard using their chosen(not sure yet what they chose-assuming > ConsoleReporter) Reporter. I need to capture those metrics, transform them > to my Events API format and send it to a Kinesis Stream. > > We use Prometheus and InfluxDB in our environments for other purposes. > > Should I use the SLF4J Reporter to dump the metrics into a log file/folder > and watch that with a Kinesis Agent and transform it somehow(?) and then > send it to the Kinesis data stream ? > > TIA, > > >
