Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-06-27 Thread Viktor Somogyi-Vass
Hi Colin, Certainly there will be some interaction and good idea with that you said, I've added it to my KIP. Will start a new discussion thread and link this one. Viktor On Wed, Jun 26, 2019 at 11:39 PM Colin McCabe wrote: > Hi Viktor, > > Good point. Sorry, I should have read the KIP more c

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-06-26 Thread Colin McCabe
Hi Viktor, Good point. Sorry, I should have read the KIP more closely. It would be good to change the title of the mail thread to reflect the new title of the KIP, "Internal Partition Reassignment Batching." I do think there will be some interaction with KIP-455 here. One example is that we'

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-06-26 Thread Viktor Somogyi-Vass
Hey Colin, I think there's some confusion here so I might change the name of this. So KIP-435 is about the internal batching of reassignments (so purely a controller change) and not about client side APIs. As per this moment these kind of improvements are listed on KIP-455's future work section so

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-06-25 Thread Colin McCabe
Hi Viktor, Now that the 2.3 release is over, we're going to be turning our attention back to working on KIP-455, which provides an API for partition reassignment, and also solves the incremental reassignment problem. Sorry about the pause, but I had to focus on the stuff that was going into 2.

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-06-25 Thread Viktor Somogyi-Vass
Hi All, I have added another improvement to this, which is to limit the parallel leader movements. I think I'll soon (maybe late this week or early next) start a vote on this too if there are no additional feedback. Thanks, Viktor On Mon, Apr 29, 2019 at 1:26 PM Viktor Somogyi-Vass wrote: > Hi

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-29 Thread Viktor Somogyi-Vass
Hi Folks, I've updated the KIP with the batching which would work on both replica and partition level. To explain it briefly: for instance if the replica level is set to 2 and partition level is set to 3, then 2x3=6 replica reassignment would be in progress at the same time. In case of reassignmen

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-15 Thread Viktor Somogyi-Vass
A follow up on the batching topic to clarify my points above. Generally I think that batching should be a core feature as Colin said the controller should possess all information that are related. Also Cruise Control (or really any 3rd party admin system) might build upon this to give more holisti

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-15 Thread Viktor Somogyi-Vass
Hey Guys, I'll reply to you all in this email: @Jun: 1. yes, it'd be a good idea to add this feature, I'll write this into the KIP. I was actually thinking about introducing a dynamic config called reassignment.parallel.partition.count and reassignment.parallel.replica.count. The first property w

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-09 Thread Jason Gustafson
Hi Colin, On a related note, what do you think about the idea of storing the > reassigning replicas in > /brokers/topics/[topic]/partitions/[partitionId]/state, rather than in the > reassignment znode? I don't think this requires a major change to the > proposal-- when the controller becomes awar

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-08 Thread George Li
Hi Colin / Jason,  Reassignment should really be doing a batches.  I am not too worried about reassignment znode getting larger.  In a real production environment,  too many concurrent reassignment and too frequent submission of reassignments seemed to cause latency spikes of kafka cluster.  S

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-08 Thread Colin McCabe
Yeah, I think adding this information to LeaderAndIsr makes sense. It would be better to track "reassigningReplicas" than "originalReplicas", I think. Tracking "originalReplicas" is going to involve sending a lot more data, since most replicas in the system are not reassigning at any given poi

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-08 Thread Jason Gustafson
Hey George, For the URP during a reassignment, if the "original_replicas" is kept for > the current pending reassignment. I think it will be very easy to compare > that with the topic/partition's ISR. If all "original_replicas" are in > ISR, then URP should be 0 for that topic/partition. Yeah,

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-07 Thread Ismael Juma
Good discussion about where we should do batching. I think if there is a clear great way to batch, then it makes a lot of sense to just do it once. However, if we think there is scope for experimenting with different approaches, then an API that tools can use makes a lot of sense. They can experime

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-07 Thread Colin McCabe
Hi George, As Jason was saying, it seems like there are two directions we could go here: an external system handling batching, and the controller handling batching. I think the controller handling batching would be better, since the controller has more information about the state of the system

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-05 Thread George Li
Hi Jason / Viktor, For the URP during a reassignment,  if the "original_replicas" is kept for the current pending reassignment. I think it will be very easy to compare that with the topic/partition's ISR.  If all "original_replicas" are in ISR, then URP should be 0 for that topic/partition. 

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-05 Thread Jason Gustafson
Hi Viktor, Thanks for writing this up. As far as questions about overlap with KIP-236, I agree it seems mostly orthogonal. I think KIP-236 may have had a larger initial scope, but now it focuses on cancellation and batching is left for future work. With that said, I think we may not actually need

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-04-04 Thread Jun Rao
Hi, Viktor, Thanks for the KIP. A couple of comments below. 1. Another potential thing to do reassignment incrementally is to move a batch of partitions at a time, instead of all partitions. This may lead to less data replication since by the time the first batch of partitions have been completel

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-02-23 Thread Viktor Somogyi-Vass
Hi Harsha, As far as I understand KIP-236 it's about enabling reassignment cancellation and as a future plan providing a queue of replica reassignment steps to allow manual reassignment chains. While I agree that the reassignment chain has a specific use case that allows fine grain control over re

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

2019-02-22 Thread Harsha
Hi Viktor, There is already KIP-236 for the same feature and George made a PR for this as well. Lets consolidate these two discussions. If you have any cases that are not being solved by KIP-236 can you please mention them in that thread. We can address as part of KIP-236. Thanks, H

[DISCUSS] KIP-435: Incremental Partition Reassignment

2019-02-22 Thread Viktor Somogyi-Vass
Hi Folks, I've created a KIP about an improvement of the reassignment algorithm we have. It aims to enable partition-wise incremental reassignment. The motivation for this is to avoid excess load that the current replication algorithm implicitly carries as in that case there are points in the algo