Re: 0.8 High-level consumer commit offset question

2013-11-05 Thread Guozhang Wang
Hi Shafaq, "Yes" to your first question. For second question that depends on what would you like to do when your request's offset is not valid. This is possible, for example, if your logs gets deleted according to retention policy and your last committed offset in ZK is within that deleted range.

0.8 High-level consumer commit offset question

2013-11-05 Thread Shafaq
Hi, I wanted to control the commitOffset signal to ZK from High-level kafka consumer. This will enable to process the message by consumer and incase of failure the offset is not moved. If I do auto.commit.enable= false and then use commitallOffset Api of high-level consumer, is that way to a

Re: High Level Consumer commit offset

2013-11-05 Thread Vadim Keylis
That I can manage. Thanks so much. On Tue, Nov 5, 2013 at 6:46 AM, Neha Narkhede wrote: > Yes, it will commit offsets only for the partitions that the consumer owns. > But over time, the set of partitions that a consumer owns can change. > > Thanks, > Neha > On Nov 5, 2013 12:17 AM, "Vadim Keyli

Re: High Level Consumer commit offset

2013-11-05 Thread Neha Narkhede
Yes, it will commit offsets only for the partitions that the consumer owns. But over time, the set of partitions that a consumer owns can change. Thanks, Neha On Nov 5, 2013 12:17 AM, "Vadim Keylis" wrote: > I am using creating Consumer.createJavaConsumerConnector(kafka 0.8) for > each topic/p

Re: High Level Consumer commit offset

2013-11-05 Thread Vadim Keylis
I am using creating Consumer.createJavaConsumerConnector(kafka 0.8) for each topic/partition. Would it be safe to assume that commit offset will apply only to stream/partition managed by that connector? Thanks, Vadim On Mon, Nov 4, 2013 at 8:43 PM, Neha Narkhede wrote: > You need to set "auto

Re: High Level Consumer commit offset

2013-11-04 Thread Shafaq
Hi, I wanted to control the commitOffset signal to ZK from High-level kafka consumer. This will enable to process the message by consumer and incase of failure the offset is not moved. If I do auto.commit.enable= false and then use commitallOffset Api of high-level consumer, is that way to a

Re: High Level Consumer commit offset

2013-11-04 Thread Vadim Keylis
Thanks so much Neha. That did the trick. Thanks so much. On Mon, Nov 4, 2013 at 8:43 PM, Neha Narkhede wrote: > You need to set "auto.offset.reset"="smallest". By default, the consumer > will start consuming the latest messages. > > Thanks, > Neha > > > On Mon, Nov 4, 2013 at 4:38 PM, Guozhang W

Re: High Level Consumer commit offset

2013-11-04 Thread Neha Narkhede
You need to set "auto.offset.reset"="smallest". By default, the consumer will start consuming the latest messages. Thanks, Neha On Mon, Nov 4, 2013 at 4:38 PM, Guozhang Wang wrote: > And exceptions you saw from the broker end, in server log? > > Guozhang > > > On Mon, Nov 4, 2013 at 4:27 PM, V

Re: High Level Consumer commit offset

2013-11-04 Thread Guozhang Wang
And exceptions you saw from the broker end, in server log? Guozhang On Mon, Nov 4, 2013 at 4:27 PM, Vadim Keylis wrote: > Thanks for confirming, but that not behavior I observe. My consumer does > not commit data to kafka. It get messages sent to kafka. Once restarted I > should of gotten mes

Re: High Level Consumer commit offset

2013-11-04 Thread Vadim Keylis
Thanks for confirming, but that not behavior I observe. My consumer does not commit data to kafka. It get messages sent to kafka. Once restarted I should of gotten messages that previously received by consumer, but on contrarily I got none. Logs confirm the initial offset been as -1. What am I doi

Re: High Level Consumer commit offset

2013-11-04 Thread Guozhang Wang
That is correct. If auto.commit.enable is set to faulse, the offsets will not be committed at all unless the consumer calls the commit function explicitly. Guozhang On Mon, Nov 4, 2013 at 2:42 PM, Vadim Keylis wrote: > Good afternoon. I was under impression if auto commit set to false then >

High Level Consumer commit offset

2013-11-04 Thread Vadim Keylis
Good afternoon. I was under impression if auto commit set to false then once consumer is restarted then logs would be replayed from the beginning. Is that correct? Thanks, Vadim