committing its offsets, then the coordinator could
>>>>> send
>>>>>>>> the add partition HeartbeatResponse, after receiving the
>>>>>>>> PartitionCommittedNotification, to the consumer gaining the partition
>>>>> and
&g
g
>>>>>>> rebalanced would see any consumption pause.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So, something like:
>>>>>>>
>>>>>>> 1. 3 consumers running with 1
t;
>>>>>>> So, something like:
>>>>>>>
>>>>>>> 1. 3 consumers running with 12 partitions balanced, 4 each
>>>>>>>
>>>>>>> 2. New consumer starts and sends JoinGroupRequest to coordinator
&
>>>>>> 3. Coordinator computes rebalance with 4 consumers: each existing
>>>>>> consumer will lose a partition assigned to the new consumer
>>>>>>
>>>>>> 4. Coordinator informs all live brokers of partition reassignments
>&
existing consumers and awaits
>> >>> PartitionCommittedNotifications from consumers losing partitions.
>> >>>
>> >>> 7. Existing consumers can continue to fetch messages from correctly
>> >>> assigned partitions
>> >&
gt;> partition and then send a PartitionCommittedNotification to the
> coordinator.
> >>>
> >>> 9. As the Coordinator receives the PartitionCommittedNotification,
> for a
> >>> particular partition from an existing consumer, it sends the
> addPartition
&
fetching that partition.
>>>
>>>
>>>
>>> If a consumer drops HeartbeatRequests within a session timeout, the
>>> coordinator would inform the brokers and they would fail fetchRequests for
>>> those partitions from that consumer. There is no chance
This seems no different than
>> this sort of failure, today.
>>
>>
>>
>> Instead of a global synchronization barrier isn’t it possible to have an
>> incremental per-partition synchronization barrier? The brokers would have
>> to be aware of this. I thi
chronization barrier isn’t it possible to have an
> incremental per-partition synchronization barrier? The brokers would have
> to be aware of this. I think of it as relativistic from each acceleration
> frame of reference, which is each consumer: event horizons.
>
>
>
> Regards,
each acceleration frame
of reference, which is each consumer: event horizons.
Regards,
Rob
-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com]
Sent: Wednesday, July 16, 2014 9:20 AM
To: users@kafka.apache.org
Subject: Re: New Consumer Design
Hi Rob,
Piggy-bac
Hi Rob,
Piggy-back the rebalance partition info in HeartbeatResponse may cause
inconsistency of the partition assignments to consumers with consecutive
triggering of rebalances, since the coordinator no longer has a
synchronization barrier any more for re-compute the distribution with a
consistent
> On Jul 14, 2014, at 3:20 PM, Baran Nohutçuoğlu wrote:
>
>
>> On Jul 8, 2014, at 3:17 PM, Guozhang Wang wrote:
>>
>> Hi All,
>>
>> We have written a wiki a few weeks back proposing a single-threaded ZK-free
>> consumer client design for 0.9:
>>
>> https://cwiki.apache.org/confluence/displa
On Jul 8, 2014, at 3:17 PM, Guozhang Wang wrote:
> Hi All,
>
> We have written a wiki a few weeks back proposing a single-threaded ZK-free
> consumer client design for 0.9:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
>
> We want to share some of the
13 matches
Mail list logo
