Hi, An issue I have regarding rebalancing, is that a call to poll() triggers the JoinGroupRequest when rebalancing is in process. In cases where a consumer is streaming more than a single batch at a time, there is no opportunity to attempt to flush any consumed batches through prior to the rebalance completing. If onPartitionsRevoked would be called via a background thread, or an alive() call, there would be an opportunity for a client to hold off from calling poll, until downstream messages are flushed prior to calling poll again to trigger the Join and onPartitionsAssigned.
The current assumption appears to be that a call to poll() indicates that there are no more in-flight messages. Attempting to decouple consumer and processor threads or the streaming of multiple batches results in unavoidable redeliveries during a rebalance. Regards Simon Souter https://github.com/cakesolutions/scala-kafka-client -- *Simon Souter* Software Engineer - Team Lead Cake Solutions Limited Find out more about The Art of Possible <http://www.cakesolutions.net/> Overview videos <http://www.cakesolutions.net/software-delivery> - Check out our wide range of services Cake’s blog <http://www.cakesolutions.net/teamblogs>- Read all about the exciting technical problems we are solving Twitter <https://twitter.com/cakesolutions> - Keep up-to-date with white papers, events, user group updates and other snippets of wisdom T: 0845 6171200 *T:* (from outside UK): +44 (0)161 443 2355 *sim...@cakesolutions.net <sim...@cakesolutions.net>* www.cakesolutions.net Company registered in UK, No. 4184567 If you have received this e-mail in error please accept our apologies, destroy it immediately and it would be greatly appreciated if you notified the sender. It is your responsibility to protect your system from viruses and any other harmful code or device. We try to eliminate them from e-mails and attachments; but we accept no liability for any which remain. We may monitor or access any or all e-mails sent to us.
