Re: Lost offsets after migration to Kafka brokers v0.10.0

2016-08-16 Thread Javier Holguera
Hi Sam, Thanks for your quick answer. We are passing “offsets.storage = kafka” to clj.kafka.consumer.zk/consumer ( https://github.com/pingles/clj-kafka/blob/master/src/clj_kafka/consumer/zk.clj#L24 ) I would expect then that offsets would be stored in both Kafka and ZK (according to Kafka docume

Re: Lost offsets after migration to Kafka brokers v0.10.0

2016-08-16 Thread Sam Pegler
clj-kafka uses the old consumer API's and offset storage in ZK. If I were you I'd migrate to https://github.com/weftio/gregor which wraps the new consumer API and stores offsets in Kafka. I'm going to assume you didn't migrate ZK state based off this? __ On 16 August 2016 at 12:15, Javier Holg

Lost offsets after migration to Kafka brokers v0.10.0

2016-08-16 Thread Javier Holguera
Hi, Yesterday my company completed a “successful” migration from Kafka brokers v0.9.0.1 to Kafka 0.10.0. However the migration can’t be considered completely successfully because we accidentally lost our offsets. Fortunately our apps are designed to be able to replay from the beginning on the top