Hi Grant, Thank you for the KIP. Very well written and easy to understand.
One question I have after reading the KIP: What are we targeting by using a Lag Aware assignment assignor? Is the goal to speed up consuming all messages from a topic? If that is the case, it sounds to me that assigning partitions based on only lag information would not be enough. There are other factors, like network latency, how fast a consumer is processing data, and consumer configuration (such as fetch.max.bytes, max.partition.fetch.bytes, ...) that impact how fast a consumer is able to consume messages. For example, let's say we have a topic with 4 partitions, and the lags are 1000, 100, 10, 1 for partitions 0 to 3. If we have two consumers c1 and c2 in the group, the Lag Aware assignment will be - c1: p0, p3 (total lag of 1001) - c2: p1, p2 (total lag of 110) Now if the speed c1 is consuming is 10% of the speed c2 is consuming then the opposite assignment (c1: p1, p2 - c2: p0, p3) would be more reasonable. I hope I'm not missing something in the KIP, and sorry if I misunderstood the purpose. Thanks. --Vahid From: Grant Neale <grantne...@hotmail.com> To: "dev@kafka.apache.org" <dev@kafka.apache.org> Date: 06/18/2017 11:04 AM Subject: [DISCUSS] KIP-169 Lag-Aware Partition Assignment Strategy Hi all, I have raised a new KIP at https://cwiki.apache.org/confluence/display/KAFKA/KIP-169+-+Lag-Aware+Partition+Assignment+Strategy The corresponding JIRA is at https://issues.apache.org/jira/browse/KAFKA-5337 I look forward to your feedback. Regards, Grant Neale
