Hi Jason, Thanks for the response. The additional timeout is certainly a welcome addition. The issue with the onPartitionsRevoked() mechanism is that it is driven through a call to poll() which works well when consuming from a single thread in a loop. We consume via a event driven wrapper around the Java client that decouples polling from message processing and can have more than one batch in-flight through an async pipeline. It is not easy to orchestrate an attempt the flush and commit of in-flight messages during a call to onPartitionsRevoked in this case. Some mechanism to allow a client to indicate a flush is complete and is ready to rejoin that is not inferred from a call to poll() would help.
Maybe this is out of scope of this KIP as described, but some of the rejected alternatives would appear to address the async use case: *Add a separate API the user can call to indicate liveness* This would be preferable from my point of view. Our calls to the Kafka driver and driven though an Akka message dispatcher (lightweight thread). We can drive hundreds of Kafka Consumers from a handful of threads. Adding a mandatory blocking thread to consumer would be a step backwards from our point of view. It would be desirable to provide both options. *Move rebalancing to the background thread instead of heartbeats only?* This would be the ideal solution to support asynchronous event driven streaming from the driver. It would be great to be able to optionally handle rebalance events in the current way or choose to handle them in via health check's thread with the ability to specify precisely when to rejoin. It appears that the Kafka contributors are aware of these concerns as they have been discussed on various wikis and mailing lists, not to mention being raised by the Kafka streams and Akka Reactive Kafka devs. I would love to see a KIP to tackle async streaming use cases specifically. We have already managed to provide a pretty solid async wrapper around the driver as-is, although there is some pretty tricky logic required to do so based on the current interface and in some cases we cant totally avoid redeliveries when rebalancing. Regards, *Simon* *Souter* https://github.com/cakesolutions/scala-kafka-client On 19 July 2016 at 02:22, Jason Gustafson <ja...@confluent.io> wrote: > Hey Simon, > > Sorry for the late response. The onPartitionsRevoked() hook is called > before the rebalance begins (that is, before the JoinGroup is sent) and is > intended to be used to flush uncommitted data and to commit corresponding > offsets. One of the main purposes of the KIP is to decouple the time > allowed to complete this from the session timeout, which now is used only > to detect failed or unreachable processes. This should give you more time > in onPartitionsRevoked() to cleanup existing state without sacrificing > failure detection. Does that make sense? > > Thanks, > Jason > > On Tue, Jul 12, 2016 at 2:57 AM, Simon Souter <sim...@cakesolutions.net> > wrote: > > > Hi, > > > > An issue I have regarding rebalancing, is that a call to poll() triggers > > the JoinGroupRequest when rebalancing is in process. In cases where a > > consumer is streaming more than a single batch at a time, there is no > > opportunity to attempt to flush any consumed batches through prior to the > > rebalance completing. If onPartitionsRevoked would be called via a > > background thread, or an alive() call, there would be an opportunity for > a > > client to hold off from calling poll, until downstream messages are > flushed > > prior to calling poll again to trigger the Join and onPartitionsAssigned. > > > > The current assumption appears to be that a call to poll() indicates that > > there are no more in-flight messages. Attempting to decouple consumer > and > > processor threads or the streaming of multiple batches results in > > unavoidable redeliveries during a rebalance. > > > > Regards > > > > Simon Souter > > > > https://github.com/cakesolutions/scala-kafka-client > > > > > > -- > > > > *Simon Souter* > > > > Software Engineer - Team Lead > > Cake Solutions Limited > > > > > > Find out more about The Art of Possible <http://www.cakesolutions.net/> > > > > Overview videos <http://www.cakesolutions.net/software-delivery> - Check > > out our wide range of services > > > > Cake’s blog <http://www.cakesolutions.net/teamblogs>- Read all about > the > > exciting technical problems we are solving > > > > Twitter <https://twitter.com/cakesolutions> - Keep up-to-date with white > > papers, events, user group updates and other snippets of wisdom > > > > T: 0845 6171200 > > > > *T:* (from outside UK): +44 (0)161 443 2355 > > > > > > *sim...@cakesolutions.net <sim...@cakesolutions.net>* > > > > www.cakesolutions.net > > > > Company registered in UK, No. 4184567 > > > > If you have received this e-mail in error please accept our apologies, > > destroy it immediately and it would be greatly appreciated if you > notified > > the sender. It is your responsibility to protect your system from viruses > > and any other harmful code or device. We try to eliminate them from > e-mails > > and attachments; but we accept no liability for any which remain. We may > > monitor or access any or all e-mails sent to us. > > > -- *Simon Souter* Software Engineer - Team Lead Cake Solutions Limited Find out more about The Art of Possible <http://www.cakesolutions.net/> Overview videos <http://www.cakesolutions.net/software-delivery> - Check out our wide range of services Cake’s blog <http://www.cakesolutions.net/teamblogs>- Read all about the exciting technical problems we are solving Twitter <https://twitter.com/cakesolutions> - Keep up-to-date with white papers, events, user group updates and other snippets of wisdom T: 0845 6171200 *T:* (from outside UK): +44 (0)161 443 2355 *sim...@cakesolutions.net <sim...@cakesolutions.net>* www.cakesolutions.net Company registered in UK, No. 4184567 If you have received this e-mail in error please accept our apologies, destroy it immediately and it would be greatly appreciated if you notified the sender. It is your responsibility to protect your system from viruses and any other harmful code or device. We try to eliminate them from e-mails and attachments; but we accept no liability for any which remain. We may monitor or access any or all e-mails sent to us.
