Hi Jorge, Thanks for the KIP! I have the same confusion with the e2e-latency metrics as Sagar above. "e2e" would seem to indicate the latency between when the record was written to Kafka and when the record was written to the sink system by the connector - however, as per the KIP it looks like it will be the latency between when the record was written to Kafka and when the record is returned by a sink task's consumer's poll? I think that metric will be a little confusing to interpret. One more thing - I was wondering if there's a particular reason for having a min metric for e2e latency but not for convert time?
Thanks, Yash On Thu, Sep 1, 2022 at 8:59 PM Sagar <sagarmeansoc...@gmail.com> wrote: > Hi Jorge, > > Thanks for the KIP. It looks like a very good addition. I skimmed through > once and had a couple of questions => > > 1) I am assuming the new metrics would be task level metric. Could you > specify the way it's done for other sink/source connector? > 2) I am slightly confused about the e2e latency metric. Let's consider the > sink connector metric. If I look at the way it's supposed to be calculated, > i.e the difference between the record timestamp and the wall clock time, it > looks like a per record metric. However, the put-batch time measures the > time to put a batch of records to external sink. So, I would assume the 2 > can't be added as is to compute the e2e latency. Maybe I am missing > something here. Could you plz clarify this. > > Thanks! > Sagar. > > On Tue, Aug 30, 2022 at 8:43 PM Jorge Esteban Quilcate Otoya < > quilcate.jo...@gmail.com> wrote: > > > Hi all, > > > > I'd like to start a discussion thread on KIP-864: Add End-To-End Latency > > Metrics to Connectors. > > This KIP aims to improve the metrics available on Source and Sink > > Connectors to measure end-to-end latency, including source and sink > record > > conversion time, and sink record e2e latency (similar to KIP-613 for > > Streams). > > > > The KIP is here: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-864%3A+Add+End-To-End+Latency+Metrics+to+Connectors > > > > Please take a look and let me know what you think. > > > > Cheers, > > Jorge. > > >
