Hi Jordan, Thanks for the KIP. I'm curious about a possible alternative where the consumer lag for the source connector can be monitored instead of the newly-proposed metric in the KIP. Although sink tasks can't directly report the successful write of a record to the sink system, they are responsible for indirectly monitoring and communicating this in the form of the offsets returned from the SinkTask::preCommit method. This should mean that, for any well-behaved connector that returns accurate offsets from its preCommit method (including connectors that perform synchronous writes in SinkTask::put, which in most cases will not override the default behavior of the preCommit method and will allow the most up-to-date offsets read from each topic to be committed to the consumer), the consumer lag for the connector should be a decent way to monitor latency. Of course, it'll be at the mercy of the commit interval for the connector and whether the connector can successfully commit offsets with its consumer, but since that often dictates where tasks will resume from if restarted, there's still plenty of value in this metric.
Cheers, Chris On Thu, Sep 2, 2021 at 7:03 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Thanks Jordan, this is a major blindspot today. > > Ryanne > > > On Wed, Sep 1, 2021, 6:03 PM Jordan Bull <jordangb...@gmail.com> wrote: > > > Hi all, > > > > I would like to start the discussion for KIP-767 involving adding latency > > metrics to Connect. The KIP can be found at > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-767%3A+Connect+Latency+Metrics > > > > Thanks, > > Jordan > > >
