Hi Gyula, Thanks for bringing this up. It is a useful addition to have a Kafka metrics reporter. I understand that we already have Prometheus and DataDog reporters in the Flink main repo. However, personally speaking, I would slightly prefer to have the Kafka metrics reporter as an ecosystem project instead of in the main repo due to the following reasons:
1. To keep core Flink more focused. So in general if a component is more relevant to external system rather than Flink, it might be good to keep it as an ecosystem project. And metrics reporter seems a good example of that. 2. This helps encourage more contributions to Flink ecosystem instead of giving the impression that anything in Flink ecosystem must be in Flink main repo. 3. To facilitate our ecosystem project authors, we have launched a website[1] to help the community keep track of and advertise the ecosystem projects. It looks a good place to put the Kafka metrics reporter. Regarding the message format, while I think use JSON by default is fine as it does not introduce much external dependency, I wonder if we should make the message format pluggable. Many companies probably already have their own serde format for all the Kafka messages. For example, maybe they would like to just use an Avro record for their metrics instead of introducing a new JSON format. Also in many cases, there could be a lot of metric messages sent by the Flink jobs. JSON format is less efficient and might have too much overhead in that case. Thanks, Jiangjie (Becket) Qin [1] https://flink-packages.org/ On Mon, Nov 18, 2019 at 3:30 AM Konstantin Knauf <konstan...@ververica.com> wrote: > Hi Gyula, > > thank you for proposing this. +1 for adding a KafkaMetricsReporter. In > terms of the dependency we could go a similar route as for the "universal" > Flink Kafka Connector which to my knowledge always tracks the latest Kafka > version as of the Flink release and relies on compatibility of the > underlying KafkaClient. JSON sounds good to me. > > Cheers, > > Konstantin > > > > > > On Sun, Nov 17, 2019 at 1:46 PM Gyula Fóra <gyf...@apache.org> wrote: > > > Hi all! > > > > Several users have asked in the past about a Kafka based metrics reporter > > which can serve as a natural connector between arbitrary metric storage > > systems and a straightforward way to process Flink metrics downstream. > > > > I think this would be an extremely useful addition but I would like to > hear > > what others in the dev community think about it before submitting a > proper > > proposal. > > > > There are at least 3 questions to discuss here: > > > > > > *1. Do we want the Kafka metrics reporter in the Flink repo?* As it is > > much more generic than other metrics reporters already included, I would > > say yes. Also as almost everyone uses Flink with Kafka it would be a > > natural reporter choice for a lot of users. > > *2. How should we handle the Kafka dependency of the connector?* > > I think it would be an overkill to add different Kafka versions here, > > so I would use Kafka 2.+ which has the best compatibility and is future > > proof > > *3. What message format should we use?* > > I would go with JSON for readability and compatibility > > > > There is a relevant JIRA open for this already. > > https://issues.apache.org/jira/browse/FLINK-14531 > > > > We at Cloudera also promote this as a scalable way of pushing metrics to > > other systems so we are very happy to contribute an implementation or > > cooperate with others on building it. > > > > Please let me know what you think! > > > > Cheers, > > Gyula > > > > > -- > > Konstantin Knauf | Solutions Architect > > +49 160 91394525 > > > Follow us @VervericaData Ververica <https://www.ververica.com/> > > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Tony) Cheng >
