RE: Dropping messages ?

2014-02-13 Thread Kat Walker
> > Jun > > > On Tue, Feb 11, 2014 at 9:43 PM, Kat Walker wrote: > > > > Are you committing offsets manually? > > > > We are not committing offsets manually. As indicated in my first email, we > > are using the high level consumer API. > > &

RE: Dropping messages ?

2014-02-11 Thread Kat Walker
ize that some messages > are lost? Do you log every message returned to Kafka consumer client? Is it > possible that a message is returned to the consumer, but is lost in the > application logic? > > Thanks, > > Jun > > > On Mon, Feb 10, 2014 at 10:23 PM, Kat

RE: Dropping messages ?

2014-02-11 Thread Kat Walker
count messages. > > Guozhang > > > > On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker wrote: > > > Hi Jun/Guozhang > > > > We might have to retry our QA tests in its entirety. We simply cannot > > reset consumer offset as there is a lot of processing

RE: Dropping messages ?

2014-02-10 Thread Kat Walker
Hi Jun/Guozhang We might have to retry our QA tests in its entirety. We simply cannot reset consumer offset as there is a lot of processing involved after consuming those messages. This might take almost a week. The Kafka message also contains `volatile` data which is fetched from a database an