Hi Rohit,
Since "records-lag-max" is defined as the Max stat, it is initialized as
-infinity, and will be updated only as "max(current-value, watermark - last
fetched offset)" when the fetcher does return some data. If you always see
"-infinity" then it seems the job started at the end offsets and hence does
not have not fetched records.
Guozhang
On Wed, Aug 31, 2016 at 2:54 PM, Rohit Valsakumar <rvalsaku...@tivo.com>
wrote:
> Hi Guozhang,
>
> I verified that the metric is available to the custom reporter I
> registered with the Kafka Streams configuration and the class does an
> asynchronous poll on the metric periodically to get it’s value and logs
> it. Once the job has caught up to the end offsets in the input topic, the
> value of the metric comes back as ‘-Infinity’. Is this expected I would
> have expected something like ‘0’.
>
> The log line in the async poller:
>
> MetricName name = metric.metricName();
>
> gLogger.info("{}, {}, {}", name.group(), name.name(), metric.value());
>
>
> And the output:
>
> consumer-fetch-manager-metrics, records-lag-max, -Infinity
>
>
> Thanks,
> Rohit
>
>
>
>
> On 8/31/16, 1:20 PM, "Rohit Valsakumar" <rvalsaku...@tivo.com> wrote:
>
> >I will take a look at it.
> >
> >Thanks,
> >Rohit
> >
> >On 8/30/16, 6:10 PM, "Guozhang Wang" <wangg...@gmail.com> wrote:
> >
> >>Hi Rohit,
> >>
> >>Just for clarification, as stated in the java doc, metricChange "is
> >>called
> >>whenever a metric is updated or added". It is not the function when a
> >>metric is recorded; in fact, the metrics collection is in the "pull"
> >>model,
> >>where the implemented reporters can fetch the current calculated values
> >>based on its metric type (rate, min, max, histogram, etc). You may want
> >>to
> >>first take a look at the default JMXReporter implementation with the "
> >>getAttribute" function which does the pulling to make sure your
> >>customized
> >>reporter does the right behavior.
> >>
> >>
> >>
> >>Guozhang
> >>
> >>On Tue, Aug 30, 2016 at 6:01 PM, Guozhang Wang <wangg...@gmail.com>
> >>wrote:
> >>
> >>> Hello Rohit,
> >>>
> >>> As you are already aware, monitoring kafka streams is no difference
> >>>than
> >>> monitoring kafka producers / consumers. So you can just monitor on its
> >>> embedded consumer's "records-lag-max" metric, which gets recorded
> >>> whenever the consumer received the fetch response.
> >>>
> >>> As for your application, your way of passing the class name of your
> >>> implemented MetricsReporter through the StreamsConfig is correct. Have
> >>> you seen records being processing by your streams application, meaning
> >>>that
> >>> there are indeed some fetched records from the fetch response?
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Mon, Aug 29, 2016 at 10:48 AM, Rohit Valsakumar
> >>><rvalsaku...@tivo.com>
> >>> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> Any opinions about monitoring the records-lag-max for a kafka streams
> >>>>job?
> >>>>
> >>>> Thanks,
> >>>> Rohit
> >>>>
> >>>> On 8/26/16, 2:53 PM, "Rohit Valsakumar" <rvalsaku...@tivo.com> wrote:
> >>>>
> >>>> >Hi all,
> >>>> >
> >>>> >I want to monitor the max lag of a kafka streams job which is
> >>>>consuming
> >>>> >from three topics and to do that I have implemented the
> >>>>MetricsReporter
> >>>> >interface which I pass through the Streams Configuration to the
> >>>> >KafkaStreams object. In the implementation¹s metricChange() method I
> >>>> >have logging for the metric with group consumer-fetch-manager-metrics
> >>>>and
> >>>> >name records-lag-max. However, the implementation never receives a
> >>>> >metricChange notification for 'records-lag-max¹.
> >>>> >
> >>>> >I would like to know if what I am doing makes sense, is the correct
> >>>> >approach and whether the implementation should be getting
> >>>>notifications
> >>>> >for 'records-lag-max¹ periodically or only when the max lag is
> >>>>changing
> >>>> >i.e. Increasing/decreasing.
> >>>> >
> >>>> >Thanks,
> >>>> >Rohit
> >>>> >
> >>>> >
> >>>> >________________________________
> >>>> >
> >>>> >This email and any attachments may contain confidential and
> >>>>privileged
> >>>> >material for the sole use of the intended recipient. Any review,
> >>>>copying,
> >>>> >or distribution of this email (or any attachments) by others is
> >>>> >prohibited. If you are not the intended recipient, please contact the
> >>>> >sender immediately and permanently delete this email and any
> >>>>attachments.
> >>>> >No employee or agent of TiVo Inc. is authorized to conclude any
> >>>>binding
> >>>> >agreement on behalf of TiVo Inc. by email. Binding agreements with
> >>>>TiVo
> >>>> >Inc. may only be made by a signed written agreement.
> >>>>
> >>>>
> >>>> ________________________________
> >>>>
> >>>> This email and any attachments may contain confidential and privileged
> >>>> material for the sole use of the intended recipient. Any review,
> >>>>copying,
> >>>> or distribution of this email (or any attachments) by others is
> >>>>prohibited.
> >>>> If you are not the intended recipient, please contact the sender
> >>>> immediately and permanently delete this email and any attachments. No
> >>>> employee or agent of TiVo Inc. is authorized to conclude any binding
> >>>> agreement on behalf of TiVo Inc. by email. Binding agreements with
> >>>>TiVo
> >>>> Inc. may only be made by a signed written agreement.
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >>
> >>
> >>--
> >>-- Guozhang
> >
> >
> >________________________________
> >
> >This email and any attachments may contain confidential and privileged
> >material for the sole use of the intended recipient. Any review, copying,
> >or distribution of this email (or any attachments) by others is
> >prohibited. If you are not the intended recipient, please contact the
> >sender immediately and permanently delete this email and any attachments.
> >No employee or agent of TiVo Inc. is authorized to conclude any binding
> >agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> >Inc. may only be made by a signed written agreement.
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>
--
-- Guozhang
