Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-12-08 Thread Colin McCabe
> > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > If I don't see additional feedback in the next few days, I'll > > start a > > > > > > > vote

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-12-03 Thread Edoardo Comar
gt; Hi all, > > > > > > > > > > > > > > > > I've updated the KIP to reflect the latest discussions. > > > > > > > > > > > > > > > > Tom, > > > > > > > > 2) Updated > > > &

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-12-02 Thread Colin McCabe
gt; > > > reasons in the Rejected Alternatives section > > > > > > > > > > > > > > Please take a look and let me know if you have any feedback. > > > > > > > Thanks > > > > > > > > > > > > >

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-11-26 Thread Mickael Maison
; > > > > > > > > > > Thanks for the feedback. > > > > > > > We also need to assign replicas when adding partitions to an > > > > > > > existing > > > > > > > topic. This is why I choose to use a list of partition ids. >

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-11-24 Thread Gwen Shapira
d the starting partition id. > > > > > > > > > > > > Let me know if you have more questions > > > > > > > > > > > > On Tue, Oct 6, 2020 at 2:16 AM Efe Gencer > > > > > > > wrote: > > &g

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-11-19 Thread Mickael Maison
; > > Hi Mickael, > > > > > > > > > > > > Thanks for the KIP! > > > > > > A call to an external system, e.g. Cruise Control, in the > > > implementation of the provided interface can indeed help with the initial >

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-11-16 Thread Gwen Shapira
> > > > > A call to an external system, e.g. Cruise Control, in the > > implementation of the provided interface can indeed help with the initial > > assignment of partitions. > > > > > > > > > > I am curious why the proposed > > `Re

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-11-16 Thread David Jacot
curious why the proposed > `ReplicaAssignor#assignReplicasToBrokers` receives a list of partition ids > as opposed to the number of partitions to create the topic with? > > > > > > > > Would you clarify if this API is expected to be used (1) only for > new topics or (2) also f

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-11-16 Thread Mickael Maison
tion ids as opposed to the number of partitions > > > to create the topic with? > > > > > > Would you clarify if this API is expected to be used (1) only for new > > > topics or (2) also for existing topics? > > > > > > Best, > > > Efe > > > _

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-11-05 Thread Mickael Maison
c with? > > > > Would you clarify if this API is expected to be used (1) only for new > > topics or (2) also for existing topics? > > > > Best, > > Efe > > ____ > > From: Mickael Maison > > Sent: Thursday, October 1, 202

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-10-06 Thread Mickael Maison
Best, > Efe > > From: Mickael Maison > Sent: Thursday, October 1, 2020 9:43 AM > To: dev > Subject: Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor > > Thanks Tom for the feedback! > > 1. If the data returned by the Rep

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-10-05 Thread Efe Gencer
Subject: Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor Thanks Tom for the feedback! 1. If the data returned by the ReplicaAssignor implementation does not match that was requested, we'll also throw a ReplicaAssignorException 2. Good point, I'll update the KIP 3. The KIP mentions an

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-10-01 Thread Mickael Maison
Thanks Tom for the feedback! 1. If the data returned by the ReplicaAssignor implementation does not match that was requested, we'll also throw a ReplicaAssignorException 2. Good point, I'll update the KIP 3. The KIP mentions an error code associated with ReplicaAssignorException: REPLICA_ASSIGNO

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-09-21 Thread Tom Bentley
Hi Mickael, A few thoughts about the ReplicaAssignor contract: 1. What happens if a ReplicaAssignor impl returns a Map where some assignments don't meet the given replication factor? 2. Fixing the signature of assignReplicasToBrokers() as you have would make it hard to pass extra information in t

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-09-11 Thread Robert Barrett
Thanks Mickael, I think adding the new Exception resolves my concerns. On Thu, Sep 3, 2020 at 9:47 AM Mickael Maison wrote: > Thanks Robert and Ryanne for the feedback. > > ReplicaAssignor implementations can throw an exception to indicate an > assignment can't be computed. This is already what

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-09-03 Thread Mickael Maison
Thanks Robert and Ryanne for the feedback. ReplicaAssignor implementations can throw an exception to indicate an assignment can't be computed. This is already what the current round robin assignor does. Unfortunately at the moment, there are no generic error codes if it fails, it's either INVALID_

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-08-28 Thread Ryanne Dolan
Thanks Mickael, the KIP makes sense to me, esp for cases where an external system (like cruise control or an operator) knows more about the target cluster state than the broker does. Ryanne On Thu, Aug 20, 2020, 10:46 AM Mickael Maison wrote: > Hi, > > I've created KIP-660 to make the replica a

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-08-28 Thread Robert Barrett
Hi Mickael, Thanks for the KIP! One question I have is around failure cases. Are ReplicaAssignor implementations expected to always compute an assignment, or is it possible for them to have unsatisfiable conditions? One example I can think of is a requirement that at least one partition be placed

[DISCUSS] KIP-660: Pluggable ReplicaAssignor

2020-08-20 Thread Mickael Maison
Hi, I've created KIP-660 to make the replica assignment logic pluggable. https://cwiki.apache.org/confluence/display/KAFKA/KIP-660%3A+Pluggable+ReplicaAssignor Please take a look and let me know if you have any feedback. Thanks