Consumer doesn't start picking existing messages from topic until new data is added

[Tarang Dawer]

Hi All

I am facing issues with the kafka 0.8 consumer with the consumer not
picking existing messages in the topic for consumption.



Following is a snippet from my code  : -


public void run() {
            ConsumerIterator<byte[],byte[]> it = stream.iterator();
            System.out.println("************************In Run : before
while loop");   //*1st SYSO*
            while (it.hasNext()) {
                byte[] message = it.next().message();
                System.out.println("Got New Mesage , Processing .. Dump in
hbase table....********************************");   //*2nd SYSO*
}


Steps: -
1. Cleaned  zookeeper and kafka logs.
2. Started both again and produced 1000 messages in the topic.
3. Started the consumer.  the consumer , i got log till  *1st SYSO* but no
progress. it kept on waiting for the stream to get messages
4. Produced 1000 more messages in the topic, now the data consumption
starts , but the messages which i produced earlier in the topic were not
consumed, as the hbase table showed only 1000 messages , which increased
further depending on the additional number of messages i produced further,
but however, the 1st initial deficit is still there.

As it seems to me, the 1st 1000 messages are being lost, although the topic
is getting created in the kafka-log directory , with the log file showing
the initial messages there in the logbefore i start the consumer) , but the
consumer does not seems to be picking them up.

I am using Zookeeper-3.4.5 .

Kafka-Settings are default , except that i the partitions for each topic
are set to 1.

Please help me out.


Thanks
Tarang Dawer