Re: Kafka 0.10 stores offsets in ZooKeeper instead of a Kafka topic

2016-10-13 Thread Samy Dindane
, "Ben Davison" wrote: I *think* Spark 2.0.0 has a Kafka 0.8 consumer, which would still use the old Zookeeper method. The use the new consumer offsets the consumer needs to be atleast Kafka 0.9 compatible. On Thu, Oct 13, 2016 at 1:55 PM, Samy Dindane wrote: Hi, I use Kafka 0.10 wit

Kafka 0.10 stores offsets in ZooKeeper instead of a Kafka topic

2016-10-13 Thread Samy Dindane
Hi, I use Kafka 0.10 with ZK 3.4.6 and my consumers' offsets aren't stored in the __consumer_offsets topic but in ZK instead. That happens whether I let the consumer commit automatically, or commit manually with enable.auto.commit set to false. Same behavior with `offsets.storage=kafka`, which