Re: New consumer - offset one gets in poll is not offset one is supposed to commit

2015-07-24 Thread Stevo Slavić
Sorry, wrong ML. On Fri, Jul 24, 2015 at 7:07 PM, Cody Koeninger wrote: > Are you intending to be mailing the spark list or the kafka list? > > On Fri, Jul 24, 2015 at 11:56 AM, Stevo Slavić wrote: > >> Hello Cody, >> >> I'm not sure we're talking about same thing. >> >> Since you're mentioning

Re: New consumer - offset one gets in poll is not offset one is supposed to commit

2015-07-24 Thread Cody Koeninger
Are you intending to be mailing the spark list or the kafka list? On Fri, Jul 24, 2015 at 11:56 AM, Stevo Slavić wrote: > Hello Cody, > > I'm not sure we're talking about same thing. > > Since you're mentioning streams I guess you were referring to current high > level consumer, while I'm talkin

Re: New consumer - offset one gets in poll is not offset one is supposed to commit

2015-07-24 Thread Stevo Slavić
Hello Cody, I'm not sure we're talking about same thing. Since you're mentioning streams I guess you were referring to current high level consumer, while I'm talking about new yet unreleased high level consumer. Poll I was referring to is https://github.com/apache/kafka/blob/trunk/clients/src/ma

Re: New consumer - offset one gets in poll is not offset one is supposed to commit

2015-07-24 Thread Cody Koeninger
Well... there are only 2 hard problems in computer science: naming things, cache invalidation, and off-by-one errors. The direct stream implementation isn't asking you to "commit" anything. It's asking you to provide a starting point for the stream on startup. Because offset ranges are inclusive

New consumer - offset one gets in poll is not offset one is supposed to commit

2015-07-24 Thread Stevo Slavić
Hello Apache Kafka community, Say there is only one topic with single partition and a single message on it. Result of calling a poll with new consumer will return ConsumerRecord for that message and it will have offset of 0. After processing message, current KafkaConsumer implementation expects o