Hi All, One of the persistent problems we see with the new consumer is the use of the session timeout in order to ensure progress. Whenever there is a delay in message processing which exceeds the session timeout, no heartbeats can be sent and the consumer is removed from the group. We seem to hit this problem everywhere the consumer is used (including Kafka Connect and Kafka Streams) and we don't always have a great solution. I've written a KIP to address this problem here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread. Have a look and let me know what you think.
Thanks, Jason
