thanks, I will definitely put this in,
does the console-producer send compressed messages by default? I haven't
specified compression for it, so assumed that it will send plain text.




On Thu, Jan 9, 2014 at 10:14 PM, Chris Curtin <curtin.ch...@gmail.com>wrote:

> If you look at the example simple consumer:
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>
> You'll see:
>
>   if (currentOffset < readOffset) {
>         System.out.println("Found an old offset: " + currentOffset + "
> Expecting: " + readOffset);
>         continue;
>     }
>
> and a comment in the 'Reading the Data' part:
>
> Also note that we are explicitly checking that the offset being read is not
> less than the offset that we requested. This is needed since if Kafka is
> compressing the messages, the fetch request will return an entire
> compressed block even if the requested offset isn't the beginning of the
> compressed block. Thus a message we saw previously may be returned again.
>
> This is probably what is happening to you
>
> Chris
>
>
> On Thu, Jan 9, 2014 at 4:00 PM, Gerrit Jansen van Vuuren <
> gerrit...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm writing a custom consumer for kafka 0.8.
> > Everything works except for the following:
> >
> > a. connect, send fetch, read all results
> > b. send fetch
> > c. send fetch
> > d. send fetch
> > e. via the console publisher, publish 2 messages
> > f. send fetch :corr-id 1
> > g. read 2 messages published :offsets [10 11] :corr-id 1
> > h. send fetch :corr-id 2
> > i. read 2 messages published :offsets [10 11] :corr-id 2
> > j.  send fetch ...
> >
> > The problem is I get the messages sent twice as a response to two
> separate
> > fetch requests. The correlation id is distinct so it cannot be that I
> read
> > the response twice. The offsets of the 2 messages are are the same so
> they
> > are duplicates, and its not the producer sending the messages twice.
> >
> > Note: the same connection is kept open the whole time, and I send
> > block,receive then send again, after the first 2 messages are read, the
> > offsets are incremented and the next fetch will ask kafka to give it
> > messages from the new offsets.
> >
> > any ideas of why kafka would be sending the messages again on the second
> > fetch request?
> >
> > Regards,
> >  Gerrit
> >
>

Reply via email to