I am running a map-reduce job to queue data from HDFS into Kafka. The mappers simply open a file and senda the data to Kafka using KafkaProducer 0.9.0.1 client.
The issue I am debugging is that the Kafka Producer async send fails to `send` to brokers after sometime. I was able to capture the behavior with logs enabled. Kafka Producer starts off doing the following: 1. NetworkClient - Initiates connection with nodes 2. NetworkClient - Sends Metadata request 3. KafkaProducer - Sends records down to the rest of the system 4. RecordAccumulator - Allocates new byte message buffer 5. Sender - Creates producer requests to send the message to the brokers. Steps 3 - 5 repeats for a while. Then all of a sudden, when no more records are needed to be sent, the network client just sends metadata requests every minute. My code is waiting on all the async producer sends to complete. I am checking this with a `Future.get()`. When I look at the heap dump, I can still see byte buffers in the RecordAccumulator's BufferPool. So it looks like Kafka is not attempting to send those records anymore. Will appreciate if anyone has any insight on what's happening here. I've also attached my thread dump and producer configs. Thanks, Praveen
