Hey Simon, Sorry for the late response. The onPartitionsRevoked() hook is called before the rebalance begins (that is, before the JoinGroup is sent) and is intended to be used to flush uncommitted data and to commit corresponding offsets. One of the main purposes of the KIP is to decouple the time allowed to complete this from the session timeout, which now is used only to detect failed or unreachable processes. This should give you more time in onPartitionsRevoked() to cleanup existing state without sacrificing failure detection. Does that make sense?
Thanks, Jason On Tue, Jul 12, 2016 at 2:57 AM, Simon Souter <sim...@cakesolutions.net> wrote: > Hi, > > An issue I have regarding rebalancing, is that a call to poll() triggers > the JoinGroupRequest when rebalancing is in process. In cases where a > consumer is streaming more than a single batch at a time, there is no > opportunity to attempt to flush any consumed batches through prior to the > rebalance completing. If onPartitionsRevoked would be called via a > background thread, or an alive() call, there would be an opportunity for a > client to hold off from calling poll, until downstream messages are flushed > prior to calling poll again to trigger the Join and onPartitionsAssigned. > > The current assumption appears to be that a call to poll() indicates that > there are no more in-flight messages. Attempting to decouple consumer and > processor threads or the streaming of multiple batches results in > unavoidable redeliveries during a rebalance. > > Regards > > Simon Souter > > https://github.com/cakesolutions/scala-kafka-client > > > -- > > *Simon Souter* > > Software Engineer - Team Lead > Cake Solutions Limited > > > Find out more about The Art of Possible <http://www.cakesolutions.net/> > > Overview videos <http://www.cakesolutions.net/software-delivery> - Check > out our wide range of services > > Cake’s blog <http://www.cakesolutions.net/teamblogs>- Read all about the > exciting technical problems we are solving > > Twitter <https://twitter.com/cakesolutions> - Keep up-to-date with white > papers, events, user group updates and other snippets of wisdom > > T: 0845 6171200 > > *T:* (from outside UK): +44 (0)161 443 2355 > > > *sim...@cakesolutions.net <sim...@cakesolutions.net>* > > www.cakesolutions.net > > Company registered in UK, No. 4184567 > > If you have received this e-mail in error please accept our apologies, > destroy it immediately and it would be greatly appreciated if you notified > the sender. It is your responsibility to protect your system from viruses > and any other harmful code or device. We try to eliminate them from e-mails > and attachments; but we accept no liability for any which remain. We may > monitor or access any or all e-mails sent to us. >
