We disabled kafka consumer metrics in flink via setting register.consumer.metrics to false in KafkaSource and enabled kafka consumer integration in datadog agent as explained here <https://www.datadoghq.com/blog/monitor-kafka-with-datadog/#configure-the-agent> .
On Fri, Aug 27, 2021 at 8:44 AM Debraj Manna <subharaj.ma...@gmail.com> wrote: > Yes we are also facing the same problem and not able to find any solution. > > On Thu, Aug 26, 2021 at 5:59 PM Chesnay Schepler <ches...@apache.org> > wrote: > >> AFAIK this metric is directly forwarded from Kafka as-is; so Flink isn't >> calculating anything. >> >> I suggest to reach out to the Kafka folks. >> >> On 25/08/2021 17:23, Shilpa Shankar wrote: >> >> Hello , >> >> We have enabled DataDogHTTPReporter to fetch metrics on flink v1.13.1 >> running on kubernetes. The metric >> flink.operator.KafkaConsumer.records_lag_max >> is not displaying accurate values. It also displays 0 most of the time and >> when it does fetch a value, it seems to be wrong when I compare them with >> the Kafka lag broker metrics. >> Could you please let us know how these metrics are calculated? Are there >> any configuration changes that need to be made to support the Kafka >> Consumer metrics? >> >> # Datadog Integration >> metrics.reporter.dghttp.class: >> org.apache.flink.metrics.datadog.DatadogHttpReporter >> metrics.reporter.dghttp.apikey: xxxx >> metrics.reporter.dghttp.maxMetricsPerRequest: 1000 >> metrics.reporter.dghttp.tags: >> flink-cluster:flink-noc-cluster,data-center:lab >> >> metrics.scope.jm: flink.jobmanager >> metrics.scope.jm.job: flink.jobmanager.job >> metrics.scope.tm: flink.taskmanager >> metrics.scope.tm.job: flink.taskmanager.job >> metrics.scope.task: flink.task >> >> Thanks, >> Shilpa >> >> >>
