See, this is why I should never say anything :) The version I have right now is very limited - it only does a clone (we needed it for some hardware testing) and a leader balance (does it using partition reassignment without actually moving partitions). We have some scripts that the other SREs have written that need to be incorporated which do partition balancing by count and by size.
Let me figure out where we are going to host this, but we can probably put out the script we have in the next couple weeks. Then we can just iterate on it. -Todd > On May 24, 2015, at 8:46 PM, Henry Cai <h...@pinterest.com.INVALID> wrote: > > Todd, > > This is very promising. Do you know when will we be able to see your tools > released to public? > >> On Sun, May 24, 2015 at 7:54 PM, Todd Palino <tpal...@gmail.com> wrote: >> >> We've built tools on top of it that both build the list based on less >> information (like "clone this broker to that one") and break it down into a >> configurable number of discrete moves so it doesn't tank the cluster. >> >> And yes, I've finally started the process of departing them from the >> LinkedIn-specific tooling so we can release them to everyone else :) >> >> -Todd >> >>>> On May 24, 2015, at 7:45 PM, Henry Cai <h...@pinterest.com.INVALID> >>> wrote: >>> >>> We have a kafka cluster with 10 brokers and we are using the kafka >>> replication tool (kafka-reassign-partitions.sh) when we need to add more >>> brokers to the cluster. But this tool tends to move too many >>> topic/partitions around at the same time which causes instability. Do we >>> have an option to do it more slowly (e.g. move one topic/partition at a >>> step) or did some one build a tool on top of >> 'kafka-reassign-partitions.sh'? >>> >>> Another use case is when a broker node went down, do we have a tool to >> move >>> the topic/partitions serviced by this node to the remaining nodes (and >>> doing that in a fashion which doesn't cause too much instability)? >>
