Re: Kafka 09 consumer does not commit offsets

2017-01-09 Thread Tzu-Li (Gordon) Tai
Good to know! On January 10, 2017 at 1:06:29 PM, Renjie Liu (liurenjie2...@gmail.com) wrote: Hi, all: I used kafka connector 0.10 and the problem is fixed. I think this maybe caused by incompatible between consumer 0.9 and broker 0.10. Thanks Henri and Gordon. On Tue, Jan 10, 2017 at 4:46 AM H

Re: Kafka 09 consumer does not commit offsets

2017-01-09 Thread Renjie Liu
Hi, all: I used kafka connector 0.10 and the problem is fixed. I think this maybe caused by incompatible between consumer 0.9 and broker 0.10. Thanks Henri and Gordon. On Tue, Jan 10, 2017 at 4:46 AM Henri Heiskanen wrote: > Hi, > > We had the same problem when running 0.9 consumer against 0.10

Re: Kafka 09 consumer does not commit offsets

2017-01-09 Thread Henri Heiskanen
Hi, We had the same problem when running 0.9 consumer against 0.10 Kafka. Upgrading Flink Kafka connector to 0.10 fixed our issue. Br, Henkka On Mon, Jan 9, 2017 at 5:39 PM, Tzu-Li (Gordon) Tai wrote: > Hi, > > Not sure what might be going on here. I’m pretty certain that for > FlinkKafkaConsu

Re: Kafka 09 consumer does not commit offsets

2017-01-09 Thread Tzu-Li (Gordon) Tai
Hi, Not sure what might be going on here. I’m pretty certain that for FlinkKafkaConsumer09 when checkpointing is turned off, the internally used KafkaConsumer client will auto commit offsets back to Kafka at a default interval of 5000ms (the default value for “auto.commit.interval.ms”). Could

Re: Kafka 09 consumer does not commit offsets

2017-01-09 Thread Timo Walther
I'm not a Kafka expert but maybe Gordon (in CC) knows more. Timo Am 09/01/17 um 11:51 schrieb Renjie Liu: Hi, all: I'm using flink 1.1.3 and kafka consumer 09. I read its code and it says that the kafka consumer will turn on auto offset commit if checkpoint is not enabled. I've turned off ch

Kafka 09 consumer does not commit offsets

2017-01-09 Thread Renjie Liu
Hi, all: I'm using flink 1.1.3 and kafka consumer 09. I read its code and it says that the kafka consumer will turn on auto offset commit if checkpoint is not enabled. I've turned off checkpoint and it seems that kafka client is not committing to offsets to kafka? The offset is important for helpin