Armand created KAFKA-14878:
------------------------------
Summary: Reduce assignment data size to improve Kafka Consumer
scalability
Key: KAFKA-14878
URL: https://issues.apache.org/jira/browse/KAFKA-14878
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Armand
This issue is linked to https://issues.apache.org/jira/browse/KAFKA-7149
which describes scalability problems with Kafka streams and was resolved by
this [PR|https://github.com/apache/kafka/pull/7185].
We have the same problem with Kafka Consumer. If we scale our solution with 50
instances (each instance has 5 consumers, so a consumer group can contain 50 *
5 = 250 consumers) with 5000 partitions, the leader of the group will send
SyncGroup requests exceeding 100 MB to the group coordinator.
The only solution we found is to increase the max.message.bytes of the
__consumer_offsets topic. Is there any plan to compress requests from the
consumer in order to avoid sending too big requests over the network?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
