Hi Guozhang,
I verified that the metric is available to the custom reporter I
registered with the Kafka Streams configuration and the class does an
asynchronous poll on the metric periodically to get it’s value and logs
it. Once the job has caught up to the end offsets in the input topic, the
value of the metric comes back as ‘-Infinity’. Is this expected I would
have expected something like ‘0’.
The log line in the async poller:
MetricName name = metric.metricName();
gLogger.info("{}, {}, {}", name.group(), name.name(), metric.value());
And the output:
consumer-fetch-manager-metrics, records-lag-max, -Infinity
Thanks,
Rohit
On 8/31/16, 1:20 PM, "Rohit Valsakumar" <rvalsaku...@tivo.com> wrote:
>I will take a look at it.
>
>Thanks,
>Rohit
>
>On 8/30/16, 6:10 PM, "Guozhang Wang" <wangg...@gmail.com> wrote:
>
>>Hi Rohit,
>>
>>Just for clarification, as stated in the java doc, metricChange "is
>>called
>>whenever a metric is updated or added". It is not the function when a
>>metric is recorded; in fact, the metrics collection is in the "pull"
>>model,
>>where the implemented reporters can fetch the current calculated values
>>based on its metric type (rate, min, max, histogram, etc). You may want
>>to
>>first take a look at the default JMXReporter implementation with the "
>>getAttribute" function which does the pulling to make sure your
>>customized
>>reporter does the right behavior.
>>
>>
>>
>>Guozhang
>>
>>On Tue, Aug 30, 2016 at 6:01 PM, Guozhang Wang <wangg...@gmail.com>
>>wrote:
>>
>>> Hello Rohit,
>>>
>>> As you are already aware, monitoring kafka streams is no difference
>>>than
>>> monitoring kafka producers / consumers. So you can just monitor on its
>>> embedded consumer's "records-lag-max" metric, which gets recorded
>>> whenever the consumer received the fetch response.
>>>
>>> As for your application, your way of passing the class name of your
>>> implemented MetricsReporter through the StreamsConfig is correct. Have
>>> you seen records being processing by your streams application, meaning
>>>that
>>> there are indeed some fetched records from the fetch response?
>>>
>>> Guozhang
>>>
>>>
>>> On Mon, Aug 29, 2016 at 10:48 AM, Rohit Valsakumar
>>><rvalsaku...@tivo.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Any opinions about monitoring the records-lag-max for a kafka streams
>>>>job?
>>>>
>>>> Thanks,
>>>> Rohit
>>>>
>>>> On 8/26/16, 2:53 PM, "Rohit Valsakumar" <rvalsaku...@tivo.com> wrote:
>>>>
>>>> >Hi all,
>>>> >
>>>> >I want to monitor the max lag of a kafka streams job which is
>>>>consuming
>>>> >from three topics and to do that I have implemented the
>>>>MetricsReporter
>>>> >interface which I pass through the Streams Configuration to the
>>>> >KafkaStreams object. In the implementation¹s metricChange() method I
>>>> >have logging for the metric with group consumer-fetch-manager-metrics
>>>>and
>>>> >name records-lag-max. However, the implementation never receives a
>>>> >metricChange notification for 'records-lag-max¹.
>>>> >
>>>> >I would like to know if what I am doing makes sense, is the correct
>>>> >approach and whether the implementation should be getting
>>>>notifications
>>>> >for 'records-lag-max¹ periodically or only when the max lag is
>>>>changing
>>>> >i.e. Increasing/decreasing.
>>>> >
>>>> >Thanks,
>>>> >Rohit
>>>> >
>>>> >
>>>> >________________________________
>>>> >
>>>> >This email and any attachments may contain confidential and
>>>>privileged
>>>> >material for the sole use of the intended recipient. Any review,
>>>>copying,
>>>> >or distribution of this email (or any attachments) by others is
>>>> >prohibited. If you are not the intended recipient, please contact the
>>>> >sender immediately and permanently delete this email and any
>>>>attachments.
>>>> >No employee or agent of TiVo Inc. is authorized to conclude any
>>>>binding
>>>> >agreement on behalf of TiVo Inc. by email. Binding agreements with
>>>>TiVo
>>>> >Inc. may only be made by a signed written agreement.
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> This email and any attachments may contain confidential and privileged
>>>> material for the sole use of the intended recipient. Any review,
>>>>copying,
>>>> or distribution of this email (or any attachments) by others is
>>>>prohibited.
>>>> If you are not the intended recipient, please contact the sender
>>>> immediately and permanently delete this email and any attachments. No
>>>> employee or agent of TiVo Inc. is authorized to conclude any binding
>>>> agreement on behalf of TiVo Inc. by email. Binding agreements with
>>>>TiVo
>>>> Inc. may only be made by a signed written agreement.
>>>>
>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
>>
>>
>>--
>>-- Guozhang
>
>
>________________________________
>
>This email and any attachments may contain confidential and privileged
>material for the sole use of the intended recipient. Any review, copying,
>or distribution of this email (or any attachments) by others is
>prohibited. If you are not the intended recipient, please contact the
>sender immediately and permanently delete this email and any attachments.
>No employee or agent of TiVo Inc. is authorized to conclude any binding
>agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>Inc. may only be made by a signed written agreement.
________________________________
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
