Re: Monitoring the max lag of a kafka streams application.

Hi Rohit, Since "records-lag-max" is defined as the Max stat, it is initialized as -infinity, and will be updated only as "max(current-value, watermark - last fetched offset)" when the fetcher does return some data. If you always see "-infinity" then it seems the job started at the end offsets and

Re: Monitoring the max lag of a kafka streams application.

Hi Guozhang, I verified that the metric is available to the custom reporter I registered with the Kafka Streams configuration and the class does an asynchronous poll on the metric periodically to get it’s value and logs it. Once the job has caught up to the end offsets in the input topic, the valu

Re: Monitoring the max lag of a kafka streams application.

I will take a look at it. Thanks, Rohit On 8/30/16, 6:10 PM, "Guozhang Wang" wrote: >Hi Rohit, > >Just for clarification, as stated in the java doc, metricChange "is called >whenever a metric is updated or added". It is not the function when a >metric is recorded; in fact, the metrics collectio

Re: Monitoring the max lag of a kafka streams application.

Hi Guozhang, Thanks for confirming the approach and yes I have a Processor node to track the flow of records through the job and it shows that the job is processing the records. I am trying to simulate a slow processing job so that I can verify if the metricChange callback is called for records-la

Re: Monitoring the max lag of a kafka streams application.

Hi Rohit, Just for clarification, as stated in the java doc, metricChange "is called whenever a metric is updated or added". It is not the function when a metric is recorded; in fact, the metrics collection is in the "pull" model, where the implemented reporters can fetch the current calculated va

Re: Monitoring the max lag of a kafka streams application.

Hello Rohit, As you are already aware, monitoring kafka streams is no difference than monitoring kafka producers / consumers. So you can just monitor on its embedded consumer's "records-lag-max" metric, which gets recorded whenever the consumer received the fetch response. As for your application

Re: Monitoring the max lag of a kafka streams application.

Hi all, Any opinions about monitoring the records-lag-max for a kafka streams job? Thanks, Rohit On 8/26/16, 2:53 PM, "Rohit Valsakumar" wrote: >Hi all, > >I want to monitor the max lag of a kafka streams job which is consuming >from three topics and to do that I have implemented the MetricsRe

Monitoring the max lag of a kafka streams application.

Hi all, I want to monitor the max lag of a kafka streams job which is consuming from three topics and to do that I have implemented the MetricsReporter interface which I pass through the Streams Configuration to the KafkaStreams object. In the implementation’s metricChange() method I have logg