Re: Consumer rebalancing retry settings and reconnecting after failure

Hi Guozhang, OK, I spent some time to understand a bit more how Kafka uses ZooKeeper and how sessions are handled and it seems that the change you proposed should do the job. Thanks :-) But I still think that (optional?) automatic restart of a consumer could be a good idea! ;-) M. Kind regard

Re: request.required.acks=-1 under high data volume

Hi Jay, Thanks for explaining the lag detection mechanism. I think my real concern is from the description of request.required.acks=-1 from kafka's document: "-1, which means that the producer gets an acknowledgement after all in-sync replicas have received the data. This option provides the best

Re: request.required.acks=-1 under high data volume

Hi Jun, I agree that defining acks=-1 as receiving acks from all replicas may not be a very useful option, but at least it does provide best durability, although at the cost of availability. The current definition of acks=-1 (receiving acks from all replicas in ISR) doesn't really guarantee bett

Re: request.required.acks=-1 under high data volume

Yes, it is true that if all replicas fall out of isr, ack with -1 is the same as 1. Normally, we don't expect replicas to fall out of isr though. You may want to read https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowtoreducechurnsinISR?WhendoesabrokerleavetheISR? to see how to minimize

Re: Consumer rebalancing retry settings and reconnecting after failure

Hi Michal, Restart consumer should be easy to implement in a script. The reason not implementing this function inside Kafka consumer is to avoid missing any potential issue/bugs causing consumer to stop. Guozhang On Mon, Jul 14, 2014 at 1:45 AM, Michal Michalski < michal.michal...@boxever.com>

Re: Facing issues with Kafka 0.8.1.1 and kafka-reassign-partitions.sh

That shouldn¹t do anything, however if you had the following: {"version":1, "partitions":[{"topic":"foo1","partition":0,"replicas":[1,2,3]} } And then submitted: {"version":1, "partitions":[{"topic":"foo1","partition":0,"replicas²:[2,1,3]} } The leader would be moved from 1->2 -Clark Cla

Consumers offset tracking

Hi, I was wondering how the offset tracking was written in zookeeper for kafka 0.8.1.1 In the doc, it's written : > Consumers track the maximum offset they have consumed in each partition. > This value is stored in a ZooKeeper directory > > /consumers/[group_id]/offsets/[topic]/[*broker_id-part

Re: Consumers offset tracking

The doc is not up to date. What you are seeing in ZK is what is expected. The broker information about the partition is found in: /brokers/topics/[topic]/partitions/[partition_id]/state The node data will tell you what ID is the current leader, as well as the ISR list. -Todd On 7/14/14, 12:11

Re: New Consumer Design

On Jul 8, 2014, at 3:17 PM, Guozhang Wang wrote: > Hi All, > > We have written a wiki a few weeks back proposing a single-threaded ZK-free > consumer client design for 0.9: > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design > > We want to share some of the

Re: Consumers offset tracking

Thank you for your answer :) Is someone planning to update the doc? François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communication - CS Games 2014 Jeux de Génie

Re: New Consumer Design

> On Jul 14, 2014, at 3:20 PM, Baran Nohutçuoğlu wrote: > > >> On Jul 8, 2014, at 3:17 PM, Guozhang Wang wrote: >> >> Hi All, >> >> We have written a wiki a few weeks back proposing a single-threaded ZK-free >> consumer client design for 0.9: >> >> https://cwiki.apache.org/confluence/displa

Re: Consumers offset tracking

Updated the website. Thanks, Jun On Mon, Jul 14, 2014 at 2:53 PM, François Langelier wrote: > Thank you for your answer :) > > Is someone planning to update the doc? > > > > François Langelier > Étudiant en génie Logiciel - École de Technologie Supérieure > > Capitaine