Hi all,
With KIP-881, consumer partition assignors attempt to improve locality by
matching consumer racks with partition replica racks. At the moment, we
don't rebalance when replicas change, so changes in replica racks are not
taken into account for partition assignment until the next rebalance,
Hi Artem,
Yes, clients can reset their `client.rack` config if the broker doesn't use
a rack-aware selector. This config is only useful for improving locality
with a rack-aware selector.
On Wed, Nov 30, 2022 at 10:05 PM Artem Livshits
wrote:
> I think it's reasonable for practical scenarios. I
I think it's reasonable for practical scenarios. Is it possible to turn
off rack awareness in the clients in case the broker selector plugin is not
compatible with rack-aware logic in the client?
-Artem
On Wed, Nov 30, 2022 at 2:46 AM Rajini Sivaram
wrote:
> Hi Artem,
>
> Understood your conce
Hi Artem,
Understood your concern - brokers could use a replica selector that uses
some other client metadata other than rack id to decide the preferred read
replica, or just use the leader. In that case, consumers wouldn't really
benefit from aligning partition assignment on rack ids. So the ques
I'm thinking of a case where the broker uses a plugin that does not use
rack ids to determine client locality, in that case the consumer might
arrive at the wrong decision based on rack ids.
-Artem
On Wed, Nov 23, 2022 at 3:54 AM Rajini Sivaram
wrote:
> Hi Artem,
>
> Thanks for reviewing the KI
Hi Artem,
Thanks for reviewing the KIP. The client doesn't rely on a particular
replica assignment logic in the broker. Instead, it matches the actual rack
assignment for partitions from the metadata with consumer racks. So there
is no assumption in the client implementation about the use of
RackA
Hi Rajini,
Thank you for the KIP, the KIP looks good to match RackAwareReplicaSelector
behavior that is available out-of-box. Which should probably be good
enough in practice.
>From the design perspective, though, RackAwareReplicaSelector is just one
possible plugin, in theory the broker could u
Hi, David and Rajini,
Thanks for the explanation. It makes sense to me now.
Jun
On Wed, Nov 16, 2022 at 1:44 AM Rajini Sivaram
wrote:
> Thanks David, that was my understanding as well.
>
> Regards,
>
> Rajini
>
> On Wed, Nov 16, 2022 at 8:08 AM David Jacot
> wrote:
>
> > Hi Jun,
> >
> > We do
Thanks David, that was my understanding as well.
Regards,
Rajini
On Wed, Nov 16, 2022 at 8:08 AM David Jacot
wrote:
> Hi Jun,
>
> We don't need to bump any RPC requests. The subscription is serialized
> (including its version) and included as bytes in the RPCs.
>
> Best,
> David
>
> On Tue, No
Hi Jun,
We don't need to bump any RPC requests. The subscription is serialized
(including its version) and included as bytes in the RPCs.
Best,
David
On Tue, Nov 15, 2022 at 11:42 PM Jun Rao wrote:
>
> Hi, Rajini,
>
> Thanks for the updated KIP. Just another minor comment. It would be useful
>
Hi, Rajini,
Thanks for the updated KIP. Just another minor comment. It would be useful
to list all RPC requests whose version needs to be bumped because of the
changes in ConsumerProtocolSubscription.
Jun
On Tue, Nov 15, 2022 at 3:45 AM Rajini Sivaram
wrote:
> Hi David,
>
> Sorry, I was out of
Hi David,
Sorry, I was out of office and hence the delay in responding. Thanks for
reviewing the KIP and answering Viktor's question (thanks for the review,
Viktor).
Responses below:
01) I was in two minds about adding new assignors, because as you said,
user experience is better if assignors us
Hi Viktor,
I can actually answer your question. KIP-848 already includes rack
awareness in the protocol. It is actually the other way around, this
KIP takes the idea from KIP-848 to implement it in the current
protocol in order to realize the benefits sooner. The new protocol
will take a while to
Hi Rajini,
Thanks for the KIP. I have a few questions/comments:
01. If I understood correctly, the plan is to add new assignors which
are rack aware. Is this right? I wonder if it is a judicious choice
here. The main drawback is that clients must be configured correctly
in order to get the benefi
Hi Rajini,
If I understand correctly, the client.rack config would stay supported
after KIP-848 but does it expand the scope of that KIP too with this
config? I mean that currently you propose ConsumerProtocolSubscription to
be used but this protocol won't be available and we need to transfer the
Hi Jun,
Thank you for the review. Yes, we should add rack id to Subscription, had
missed that part. Updated the KIP, thank you for pointing that out!
Regards,
Rajini
On Wed, Nov 2, 2022 at 7:06 PM Jun Rao wrote:
> Hi, Rajini,
>
> Thanks for the KIP. Just one comment.
>
> Should we add rackId
Hi, Rajini,
Thanks for the KIP. Just one comment.
Should we add rackId to GroupSubscription.Subscription for each member?
Thanks,
Jun
On Wed, Nov 2, 2022 at 4:57 AM Rajini Sivaram
wrote:
> Hi all,
>
> I have submitted KIP-881 to implement rack-aware partition assignment for
> consumers:
>
>
Hi all,
I have submitted KIP-881 to implement rack-aware partition assignment for
consumers:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers.
It adds rack id to the consumer group protocol to propagate rack
information so that rack-a
18 matches
Mail list logo
