Hi Jason, Thank you for the message. It seems quite interesting. So something I am not sure about "pause" and "resume" is it works based on a partition allocation. What will happen if more partitions are assigned to a single consumer? For example, in the case where we have over-partition a Kafka topic to reserve some capacity for scaling up in case of burst traffic.
Regards, Ali On Sat, May 9, 2020 at 11:52 PM Jason Turim <ja...@signalvine.com> wrote: > Hi Ali, > > You may want to look at using the consumer pause / resume api. Its a > mechanism that allows you to poll without retrieving new messages. > > I employed this strategy to effectively handle highly variable workloads by > processing them in a background thread. First pause the consumer when > messages were received. Then start a background thread to process the > data, meanwhile the primary thread continues to poll (it won't retrieve > data while the consumer is paused). Finally, when the bg processing is > complete, call resume on the consumer to retrieve the next batch of > messages. > > More info here in the section "Detecting Consumer Failures" - > > https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > hth, > jt > > On Fri, May 8, 2020, 10:50 PM Chris Toomey <ctoo...@gmail.com> wrote: > > > What exactly is your understanding of what's happening when you say "the > > pipeline will be blocked by the group coordinator for up to " > > max.poll.interval.ms""? Please explain that. > > > > There's no universal recipe for "long-running jobs", there's just > > particular issues you might be encountering and suggested solutions to > > those issues. > > > > > > > > On Fri, May 8, 2020 at 7:03 PM Ali Nazemian <alinazem...@gmail.com> > wrote: > > > > > Hi Chris, > > > > > > I am not sure where I said about the "automatic partition > reassignment", > > > but what I know here is the side effect of increasing " > > > max.poll.interval.ms" > > > is if the consumer hangs for whatever reason the pipeline will be > blocked > > > by the group coordinator for up to "max.poll.interval.ms". So I am not > > > sure > > > if this is because of the automatic partition assignment or something > > else. > > > What I am looking for is how I can deal with long-running jobs in > Apache > > > Kafka. > > > > > > Thanks, > > > Ali > > > > > > On Sat, May 9, 2020 at 4:25 AM Chris Toomey <ctoo...@gmail.com> wrote: > > > > > > > I interpreted your post as saying "when our consumer gets stuck, > > Kafka's > > > > automatic partition reassignment kicks in and that's problematic for > > us." > > > > Hence I suggested not using the automatic partition assignment, which > > per > > > > my interpretation would address your issue. > > > > > > > > Chris > > > > > > > > On Fri, May 8, 2020 at 2:19 AM Ali Nazemian <alinazem...@gmail.com> > > > wrote: > > > > > > > > > Thanks, Chris. So what is causing the consumer to get stuck is a > side > > > > > effect of the built-in partition assignment in Kafka and by > > overriding > > > > that > > > > > behaviour I should be able to address the long-running job issue, > is > > > that > > > > > right? Can you please elaborate more on this? > > > > > > > > > > Regards, > > > > > Ali > > > > > > > > > > On Fri, May 8, 2020 at 1:09 PM Chris Toomey <ctoo...@gmail.com> > > wrote: > > > > > > > > > > > You really have to decide what behavior it is you want when one > of > > > your > > > > > > consumers gets "stuck". If you don't like the way the group > > protocol > > > > > > dynamically manages topic partition assignments or can't figure > out > > > an > > > > > > appropriate set of configuration settings that achieve your goal, > > you > > > > can > > > > > > always elect to not use the group protocol and instead manage > topic > > > > > > partition assignment yourself. As I just replied to another post, > > > > > there's a > > > > > > nice writeup of this under "Manual Partition Assignment" in > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > > > > . > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > On Thu, May 7, 2020 at 12:37 AM Ali Nazemian < > > alinazem...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > To help understanding my case in more details, the error I can > > see > > > > > > > constantly is the consumer losing heartbeat and hence > apparently > > > the > > > > > > group > > > > > > > get rebalanced based on the log I can see from Kafka side: > > > > > > > > > > > > > > GroupCoordinator 11]: Member > > > > > > > consumer-3-f46e14b4-5998-4083-b7ec-bed4e3f374eb in group foo > has > > > > > failed, > > > > > > > removing it from the group > > > > > > > > > > > > > > Thanks, > > > > > > > Ali > > > > > > > > > > > > > > On Thu, May 7, 2020 at 2:38 PM Ali Nazemian < > > alinazem...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > With the emerge of using Apache Kafka for event-driven > > > > architecture, > > > > > > one > > > > > > > > thing that has become important is how to tune apache Kafka > > > > consumer > > > > > to > > > > > > > > manage long-running jobs. The main issue raises when we set a > > > > > > relatively > > > > > > > > large value for "max.poll.interval.ms". Setting this value > > will, > > > > of > > > > > > > > course, resolve the issue of repetitive rebalance, but > creates > > > > > another > > > > > > > > operational issue. I am looking for some sort of golden > > strategy > > > to > > > > > > deal > > > > > > > > with long-running jobs with Apache Kafka. > > > > > > > > > > > > > > > > If the consumer hangs for whatever reason, there is no easy > way > > > of > > > > > > > passing > > > > > > > > that stage. It can easily block the pipeline, and you cannot > do > > > > much > > > > > > > about > > > > > > > > it. Therefore, it came to my mind that I am probably missing > > > > > something > > > > > > > > here. What are the expectations? Is it not valid to use > Apache > > > > Kafka > > > > > > for > > > > > > > > long-live jobs? Are there any other parameters need to be > set, > > > and > > > > > the > > > > > > > > issue of a consumer being stuck is caused by > misconfiguration? > > > > > > > > > > > > > > > > I can see there are a lot of the same issues have been raised > > > > > regarding > > > > > > > > "the consumer is stuck" and usually, the answer has been > "yeah, > > > > > that's > > > > > > > > because you have a long-running job, etc.". I have seen > > different > > > > > > > > suggestions: > > > > > > > > > > > > > > > > - Avoid using long-running jobs. Read the message, submit it > > into > > > > > > another > > > > > > > > thread and let the consumer to pass. Obviously this can cause > > > data > > > > > loss > > > > > > > and > > > > > > > > it would be a difficult problem to handle. It might be better > > to > > > > > avoid > > > > > > > > using Kafka in the first place for these types of requests. > > > > > > > > > > > > > > > > - Avoid using apache Kafka for long-running requests > > > > > > > > > > > > > > > > - Workaround based approaches like if the consumer is > blocked, > > > try > > > > to > > > > > > use > > > > > > > > another consumer group and set the offset to the current > value > > > for > > > > > the > > > > > > > new > > > > > > > > consumer group, etc. > > > > > > > > > > > > > > > > There might be other suggestions I have missed here, but that > > is > > > > not > > > > > > the > > > > > > > > point of this email. What I am looking for is what is the > best > > > > > practice > > > > > > > for > > > > > > > > dealing with long-running jobs with Apache Kafka. I cannot > > easily > > > > > avoid > > > > > > > > using Kafka because it plays a critical part in our > application > > > and > > > > > > data > > > > > > > > pipeline. On the other side, we have had so many challenges > to > > > keep > > > > > the > > > > > > > > long-running jobs stable operationally. So I would appreciate > > it > > > if > > > > > > > someone > > > > > > > > can help me to understand what approach can be taken to deal > > with > > > > > these > > > > > > > > jobs with Apache Kafka as a message broker. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Ali > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > A.Nazemian > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > A.Nazemian > > > > > > > > > > > > > > > > > > -- > > > A.Nazemian > > > > > > -- A.Nazemian
