Re: [DISCUSS] KIP-767 Connect Latency Metrics

2021-09-09 Thread Jordan Bull
Hey Chris, Thanks for jumping in! I have been using the consumer lag as an indicator for some time, but when measured directly at the consumer, it will not factor in the time that Connect actually spends transforming and sending the messages. This is certainly useful for measuring if the connector

Re: [DISCUSS] KIP-767 Connect Latency Metrics

2021-09-07 Thread Chris Egerton
Hi Jordan, Thanks for the KIP. I'm curious about a possible alternative where the consumer lag for the source connector can be monitored instead of the newly-proposed metric in the KIP. Although sink tasks can't directly report the successful write of a record to the sink system, they are responsi

Re: [DISCUSS] KIP-767 Connect Latency Metrics

2021-09-02 Thread Ryanne Dolan
Thanks Jordan, this is a major blindspot today. Ryanne On Wed, Sep 1, 2021, 6:03 PM Jordan Bull wrote: > Hi all, > > I would like to start the discussion for KIP-767 involving adding latency > metrics to Connect. The KIP can be found at > > https://cwiki.apache.org/confluence/display/KAFKA/KIP

[DISCUSS] KIP-767 Connect Latency Metrics

2021-09-01 Thread Jordan Bull
Hi all, I would like to start the discussion for KIP-767 involving adding latency metrics to Connect. The KIP can be found at https://cwiki.apache.org/confluence/display/KAFKA/KIP-767%3A+Connect+Latency+Metrics Thanks, Jordan