Are the offset used in the 2 fetch requests the same? If so, you will get the same messages twice. You consumer is responsible for advancing the offsets after consumption.
Thanks, Jun On Thu, Jan 9, 2014 at 1:00 PM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > Hi, > > I'm writing a custom consumer for kafka 0.8. > Everything works except for the following: > > a. connect, send fetch, read all results > b. send fetch > c. send fetch > d. send fetch > e. via the console publisher, publish 2 messages > f. send fetch :corr-id 1 > g. read 2 messages published :offsets [10 11] :corr-id 1 > h. send fetch :corr-id 2 > i. read 2 messages published :offsets [10 11] :corr-id 2 > j. send fetch ... > > The problem is I get the messages sent twice as a response to two separate > fetch requests. The correlation id is distinct so it cannot be that I read > the response twice. The offsets of the 2 messages are are the same so they > are duplicates, and its not the producer sending the messages twice. > > Note: the same connection is kept open the whole time, and I send > block,receive then send again, after the first 2 messages are read, the > offsets are incremented and the next fetch will ask kafka to give it > messages from the new offsets. > > any ideas of why kafka would be sending the messages again on the second > fetch request? > > Regards, > Gerrit >
