I am not sure off the top, but since the method is on the consumer my intuition is that it would pause all the partitions the consumer is reading from. I think the best thing to do is write a little test harness app to verify the behavior.
On Mon, May 11, 2020 at 7:31 AM Ali Nazemian <alinazem...@gmail.com> wrote: > Hi Jason, > > Thank you for the message. It seems quite interesting. So something I am > not sure about "pause" and "resume" is it works based on a partition > allocation. What will happen if more partitions are assigned to a single > consumer? For example, in the case where we have over-partition a Kafka > topic to reserve some capacity for scaling up in case of burst traffic. > > Regards, > Ali > > On Sat, May 9, 2020 at 11:52 PM Jason Turim <ja...@signalvine.com> wrote: > > > Hi Ali, > > > > You may want to look at using the consumer pause / resume api. Its a > > mechanism that allows you to poll without retrieving new messages. > > > > I employed this strategy to effectively handle highly variable workloads > by > > processing them in a background thread. First pause the consumer when > > messages were received. Then start a background thread to process the > > data, meanwhile the primary thread continues to poll (it won't retrieve > > data while the consumer is paused). Finally, when the bg processing is > > complete, call resume on the consumer to retrieve the next batch of > > messages. > > > > More info here in the section "Detecting Consumer Failures" - > > > > > https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > > hth, > > jt > > > > On Fri, May 8, 2020, 10:50 PM Chris Toomey <ctoo...@gmail.com> wrote: > > > > > What exactly is your understanding of what's happening when you say > "the > > > pipeline will be blocked by the group coordinator for up to " > > > max.poll.interval.ms""? Please explain that. > > > > > > There's no universal recipe for "long-running jobs", there's just > > > particular issues you might be encountering and suggested solutions to > > > those issues. > > > > > > > > > > > > On Fri, May 8, 2020 at 7:03 PM Ali Nazemian <alinazem...@gmail.com> > > wrote: > > > > > > > Hi Chris, > > > > > > > > I am not sure where I said about the "automatic partition > > reassignment", > > > > but what I know here is the side effect of increasing " > > > > max.poll.interval.ms" > > > > is if the consumer hangs for whatever reason the pipeline will be > > blocked > > > > by the group coordinator for up to "max.poll.interval.ms". So I am > not > > > > sure > > > > if this is because of the automatic partition assignment or something > > > else. > > > > What I am looking for is how I can deal with long-running jobs in > > Apache > > > > Kafka. > > > > > > > > Thanks, > > > > Ali > > > > > > > > On Sat, May 9, 2020 at 4:25 AM Chris Toomey <ctoo...@gmail.com> > wrote: > > > > > > > > > I interpreted your post as saying "when our consumer gets stuck, > > > Kafka's > > > > > automatic partition reassignment kicks in and that's problematic > for > > > us." > > > > > Hence I suggested not using the automatic partition assignment, > which > > > per > > > > > my interpretation would address your issue. > > > > > > > > > > Chris > > > > > > > > > > On Fri, May 8, 2020 at 2:19 AM Ali Nazemian <alinazem...@gmail.com > > > > > > wrote: > > > > > > > > > > > Thanks, Chris. So what is causing the consumer to get stuck is a > > side > > > > > > effect of the built-in partition assignment in Kafka and by > > > overriding > > > > > that > > > > > > behaviour I should be able to address the long-running job issue, > > is > > > > that > > > > > > right? Can you please elaborate more on this? > > > > > > > > > > > > Regards, > > > > > > Ali > > > > > > > > > > > > On Fri, May 8, 2020 at 1:09 PM Chris Toomey <ctoo...@gmail.com> > > > wrote: > > > > > > > > > > > > > You really have to decide what behavior it is you want when one > > of > > > > your > > > > > > > consumers gets "stuck". If you don't like the way the group > > > protocol > > > > > > > dynamically manages topic partition assignments or can't figure > > out > > > > an > > > > > > > appropriate set of configuration settings that achieve your > goal, > > > you > > > > > can > > > > > > > always elect to not use the group protocol and instead manage > > topic > > > > > > > partition assignment yourself. As I just replied to another > post, > > > > > > there's a > > > > > > > nice writeup of this under "Manual Partition Assignment" in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > > > > > . > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > > > > On Thu, May 7, 2020 at 12:37 AM Ali Nazemian < > > > alinazem...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > To help understanding my case in more details, the error I > can > > > see > > > > > > > > constantly is the consumer losing heartbeat and hence > > apparently > > > > the > > > > > > > group > > > > > > > > get rebalanced based on the log I can see from Kafka side: > > > > > > > > > > > > > > > > GroupCoordinator 11]: Member > > > > > > > > consumer-3-f46e14b4-5998-4083-b7ec-bed4e3f374eb in group foo > > has > > > > > > failed, > > > > > > > > removing it from the group > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Ali > > > > > > > > > > > > > > > > On Thu, May 7, 2020 at 2:38 PM Ali Nazemian < > > > alinazem...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > With the emerge of using Apache Kafka for event-driven > > > > > architecture, > > > > > > > one > > > > > > > > > thing that has become important is how to tune apache Kafka > > > > > consumer > > > > > > to > > > > > > > > > manage long-running jobs. The main issue raises when we > set a > > > > > > > relatively > > > > > > > > > large value for "max.poll.interval.ms". Setting this value > > > will, > > > > > of > > > > > > > > > course, resolve the issue of repetitive rebalance, but > > creates > > > > > > another > > > > > > > > > operational issue. I am looking for some sort of golden > > > strategy > > > > to > > > > > > > deal > > > > > > > > > with long-running jobs with Apache Kafka. > > > > > > > > > > > > > > > > > > If the consumer hangs for whatever reason, there is no easy > > way > > > > of > > > > > > > > passing > > > > > > > > > that stage. It can easily block the pipeline, and you > cannot > > do > > > > > much > > > > > > > > about > > > > > > > > > it. Therefore, it came to my mind that I am probably > missing > > > > > > something > > > > > > > > > here. What are the expectations? Is it not valid to use > > Apache > > > > > Kafka > > > > > > > for > > > > > > > > > long-live jobs? Are there any other parameters need to be > > set, > > > > and > > > > > > the > > > > > > > > > issue of a consumer being stuck is caused by > > misconfiguration? > > > > > > > > > > > > > > > > > > I can see there are a lot of the same issues have been > raised > > > > > > regarding > > > > > > > > > "the consumer is stuck" and usually, the answer has been > > "yeah, > > > > > > that's > > > > > > > > > because you have a long-running job, etc.". I have seen > > > different > > > > > > > > > suggestions: > > > > > > > > > > > > > > > > > > - Avoid using long-running jobs. Read the message, submit > it > > > into > > > > > > > another > > > > > > > > > thread and let the consumer to pass. Obviously this can > cause > > > > data > > > > > > loss > > > > > > > > and > > > > > > > > > it would be a difficult problem to handle. It might be > better > > > to > > > > > > avoid > > > > > > > > > using Kafka in the first place for these types of requests. > > > > > > > > > > > > > > > > > > - Avoid using apache Kafka for long-running requests > > > > > > > > > > > > > > > > > > - Workaround based approaches like if the consumer is > > blocked, > > > > try > > > > > to > > > > > > > use > > > > > > > > > another consumer group and set the offset to the current > > value > > > > for > > > > > > the > > > > > > > > new > > > > > > > > > consumer group, etc. > > > > > > > > > > > > > > > > > > There might be other suggestions I have missed here, but > that > > > is > > > > > not > > > > > > > the > > > > > > > > > point of this email. What I am looking for is what is the > > best > > > > > > practice > > > > > > > > for > > > > > > > > > dealing with long-running jobs with Apache Kafka. I cannot > > > easily > > > > > > avoid > > > > > > > > > using Kafka because it plays a critical part in our > > application > > > > and > > > > > > > data > > > > > > > > > pipeline. On the other side, we have had so many challenges > > to > > > > keep > > > > > > the > > > > > > > > > long-running jobs stable operationally. So I would > appreciate > > > it > > > > if > > > > > > > > someone > > > > > > > > > can help me to understand what approach can be taken to > deal > > > with > > > > > > these > > > > > > > > > jobs with Apache Kafka as a message broker. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Ali > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > A.Nazemian > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > A.Nazemian > > > > > > > > > > > > > > > > > > > > > > > -- > > > > A.Nazemian > > > > > > > > > > > > -- > A.Nazemian > -- Jason Turim (he, him & his) ​Vice President of Software Engineering SignalVine Inc <http://www.signalvine.com> (m) 415-407-6501
