@huqedi replying to your comment on the PR ( https://github.com/apache/kafka/pull/14077#discussion_r1314592488), quote:
"I guess we have a disagreement about lag? My understanding of lag is: the real LEO of the source cluster partition minus the LEO that has been written to the target cluster. It seems that your definition of lag is: the lag between the mirror task getting data from consumption and writing it to the target cluster?" Yes, this is the case. I've missed the fact that the consumer itself might be lagging behind the actual data in the partition. I believe your definition of the lag is more precise, but: Implementing it this way will come at the cost of an extra listOffsets request, introducing the overhead that you mentioned in your initial comment. If you have enough insights about this, what would you say is the chances of the task consumer lagging behind the LEO of the partition? Are they big enough to justify the extra call to listOffsets? @Viktor, any thoughts? Thanks, Elkhan On Mon, 4 Sept 2023 at 09:36, Elxan Eminov <elxanemino...@gmail.com> wrote: > I already have the PR for this so if it will make it easier to discuss, > feel free to take a look: https://github.com/apache/kafka/pull/14077 > > On Mon, 4 Sept 2023 at 09:17, hudeqi <16120...@bjtu.edu.cn> wrote: > >> But does the offset of the last `ConsumerRecord` obtained in poll not >> only represent the offset of this record in the source cluster? It seems >> that it cannot represent the LEO of the source cluster for this partition. >> I understand that the offset lag introduced here should be the LEO of the >> source cluster minus the offset of the last record to be polled? >> >> best, >> hudeqi >> >> >> > -----原始邮件----- >> > 发件人: "Elxan Eminov" <elxanemino...@gmail.com> >> > 发送时间: 2023-09-04 14:52:08 (星期一) >> > 收件人: dev@kafka.apache.org >> > 抄送: >> > 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 >> metric >> > >> </elxanemino...@gmail.com> > >
