Hi all, Kafka is not sending messages to consumer though the consumer is still active and subscribed to the partition. Please help us to know why this behavior is happening in kafka.
Infrastructure: Kafka is running in cluster mode with 3 brokers and 3 zookeeper instances. Kafka broker is running in t2.xlarge AWS instance. 3 Zookeeper instances is running in single t2.xlarge AWS instance. Kafka Version: 1.1.0 How to reproduce: * Continuously send 50,000 records/sec where each record size is about 200 bytes approx. for 24hour duration. * We have 25 topics with 4 partitions each. All 25 topics are consumed by 4 consumers with auto assignment of partition by kafka. * Kafka clears all records in partitions except 2 partitions. For two partitions , only 27 messages are delivered. Remaining messages has not delivered to consumer. * During the run kafka brokers cpu utilization is 5% Problematic partitions details: TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID topic-di14 3 27 3919844 3919817 consumer-7-dfaf4c6f-5b71-455e-ab12-9984446a9bf4 /10.10.9.22 consumer-7 topic-di9 0 27 12231117 12231090 consumer-22-7922babd-8f8b-49db-a111-df523962cca8 /10.10.9.22 consumer-22 I am not able to see any error in kafka as well as my consumer application. Consumer is active when i ran ./bin/kafka-consumer-groups.sh command in kafka. Kafka Consumer settings: enable.auto.commit = true session.timeout.ms = 90000 heartbeat.interval.ms = 30000 max.poll.records = 1000 poll-interval = 60ms poll-timeout = 60ms Regards, Gnana
