Thanks Luke. I’ve always passed a duration of 0 if polling while paused. The logic being that the poll is only used to preserve group membership and there’s no need to wait for anything. That probably explains why I would have been unlikely to ever see this behavior.
I agree with changing the implementation to guarantee that a pause is preserved after rebalance. That is in line with how the API has always been documented, despite the comments in the original JIRA. And if I understood correctly, this is already done for COOPERATIVE anyway. ----- Original Message ----- From: Luke Chen <dev@kafka.apache.org> To: RYAN LESLIE, dev@kafka.apache.org At: 02/16/22 22:02:41 UTC-05:00 Hi Ryan, > I've used pause() between version 0.10 and 2.7 with EAGER protocol only and have not yet observed cases where poll() returned messages unless resume() had been called. Yes, this issue only happens when group rebalance is triggered, the pause flag will be cleared. And if the Consumer#poll() duration is long enough to fetch records after rebalance completes, the records would return. @Riven, is my understanding correct? And also, as discussed with Guozhang, you can close this KIP discussion and start to implement it directly. Thank you. Luke On Tue, Feb 15, 2022 at 7:46 AM Ryan Leslie (BLOOMBERG/ 919 3RD A) < rles...@bloomberg.net> wrote: > Apologies, but as worded KIP-822 / KAFKA-13425 are a bit challenging to > follow. I've used pause() between version 0.10 and 2.7 with EAGER protocol > only and have not yet observed cases where poll() returned messages unless > resume() had been called. The KIP seems to contradict this though, and > indicate that poll() does return messages for EAGER protocol. Maybe a > summary of the conditions where this bug occurs could help? > > From: dev@kafka.apache.org At: 02/14/22 17:32:56 UTC-5:00To: > dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-822: Optimize the semantics of > KafkaConsumer#pause to be consistent between the two RebalanceProtocols > > Hi Luke, > > Yeah my understanding of the KIP are from > https://issues.apache.org/jira/browse/KAFKA-13463 and > https://issues.apache.org/jira/browse/KAFKA-13425, and I think we and > Riven are on the same page. > > If the changes are just within the internal classes, I think we can just > treat it as a normal bug fix and not need a KIP for it. > > > Guozhang > > On Mon, Feb 14, 2022 at 2:22 AM Luke Chen <show...@gmail.com> wrote: > > > Hi Guozhang, > > > > In short, what Riven wants to do, is that we keep the `pause` flag after > > rebalanced, while before this change, Rebalance does not preserve > > pause/resume state as described in KAFKA-2350 > > <https://issues.apache.org/jira/browse/KAFKA-2350>. > > In this case, do you think we can skip the KIP and move on the > > implementation? > > > > I'm good if we need a KIP or not, just want to make it clear in case it > > breaks any existing expectation/behavior. > > > > Thank you. > > Luke > > > > On Mon, Feb 14, 2022 at 10:09 AM Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Hello Riven, > > > > > > > > > Thanks for bringing this proposal. As we discussed on the JIRA I'm > > > personally in favor of this fix. But if all the proposed changes are in > > > `ConsumerCoordinator`, then we do not need a KIP since that class is > > > internal only. > > > > > > > > > Guozhang > > > > > > On Sat, Feb 12, 2022 at 1:35 AM Riven Sun <riven....@zoom.us.invalid> > > > wrote: > > > > > > > Sorry, I sent this email via GMail. Refer to the contents of other > > > people's > > > > DISSCUSS emails. Mistakenly introduced someone else's KIP. > > > > > > > > The KIP related to this DISCUSS is > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199534763 > > > > > > > > Thank you for your kindness > > > > RivenSun > > > > > > > > On Sat, Feb 12, 2022 at 5:32 PM Riven Sun <riven....@zoom.us> wrote: > > > > > > > > > > > > > >> Sorry, I sent this email via GMail. Refer to the contents of other > > > > >> people's DISSCUSS emails. Mistakenly introduced someone else's > KIP. > > > > >> > > > > >> The KIP related to this DISCUSS is > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199534763 > > > > >> > > > > >> Thank you for your kindness > > > > >> RivenSun > > > > >> > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > -- > -- Guozhang > > >
