Hi, This is a known issue. The 0.10 release will fix this. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records for some background.
Cheers, Jens Den ons 4 maj 2016 19:32Abhinav Solan <abhinav.so...@gmail.com> skrev: > Hi, > > I am using kafka-0.9.0.1 and have configured the Kafka consumer to fetch > 8192 bytes by setting max.partition.fetch.bytes > > Here are the properties I am using > > props.put("bootstrap.servers", servers); > props.put("group.id", "perf-test"); > props.put("offset.storage", "kafka"); > props.put("enable.auto.commit", "false"); > props.put("session.timeout.ms", 60000); > props.put("request.timeout.ms", 70000); > props.put("heartbeat.interval.ms", 50000); > props.put("auto.offset.reset", "latest"); > props.put("max.partition.fetch.bytes", "8192"); > props.put("key.deserializer", > "org.apache.kafka.common.serialization.StringDeserializer"); > props.put("value.deserializer", > "org.apache.kafka.common.serialization.StringDeserializer"); > > I am setting up 12 Consumers with 4 workers each to listen on a topic with > 200 partitions. > I have also enabled the compression when sending to Kafka. > > The problem I am getting is, even though the fetch size is less, the > consumers when polling, poll too many records. If the topics have many > messages and it is behind in the consumption it tries to fetch bigger size, > if the consumer is not behind then it try and fetch around 45, but anyways > if I set the max.partition.fetch.bytes shouldn't the fetch size have an > upper limit ? Is there any other setting I am missing here ? > I am myself controlling the message size so it's not that some bigger > messages are coming through, each message must be around 200-300 bytes > only. > > Due the large number of messages it is polling, the inner process sometimes > not able to finish the process within the heartbeat interval limit, which > makes the consumer rebalancing kick in, again and again, this only happens > when the consumer is way behind in offset e.g there are 100000 messages to > be processed in the topic. > > Thanks > -- Jens Rantil Backend Developer @ Tink Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden For urgent matters you can reach me at +46-708-84 18 32.