Hi Jason, Thanks for your response. See replies inline:
On Tuesday, December 15, 2015, Jason Gustafson <ja...@confluent.io> wrote: > Hey Jens, > > I'm not sure I understand why increasing the session timeout is not an > option. Is the issue that there's too much uncertainly about processing > time to set an upper bound for each round of the poll loop? > Yes, that's the issue. > One general workaround would be to move the processing into another thread. > For example, you could add messages to a blocking queue and have the > processor poll them. We have a pause() API which can be used to prevent > additional fetches when the queue is full. This would allow you to continue > the poll loop instead of blocking in offer(). The tricky thing would be > handling rebalances since you'd need to clear the queue and get the last > offset from the processors. Enough people probably will end up doing > something like this that we should probably work through an example on how > to do it properly. > Hm, as far as I've understood the consumer will only send heartbeats to the broker when poll() is being called. If I would call pause() on a consumer (from a separate thread) I understand poll() will block undefinitely. Will the polling consumer still send heartbeats when blocked? Or would a pause for too long (while my records are being processed) eventually lead to session timeout? If the latter, that would sort of defeat the purpose since I am trying to avoid unnecessary rebalancing of consumers when there is high pressure on the consumers. Regarding handling of rebalancing for a queue solution you describe; It really sounds very complicated. It's probably doable, but doesn't this sort of defeat the purpose of the high level consumer API? I mean, it sounds like it should gracefully handle slow consumption of varying size. I might be wrong. Thanks, Jens -- Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook <https://www.facebook.com/#!/tink.se> Linkedin <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary> Twitter <https://twitter.com/tink>
