Re: Replication tools to move topics/partitions gradually

2015-05-25 Thread Henry Cai
Thanks Todd. Let us know when these tools are announced. On Sun, May 24, 2015 at 11:00 PM, Todd Palino wrote: > See, this is why I should never say anything :) > > The version I have right now is very limited - it only does a clone (we > needed it for some hardware testing) and a leader balance

Re: Replication tools to move topics/partitions gradually

2015-05-24 Thread Todd Palino
See, this is why I should never say anything :) The version I have right now is very limited - it only does a clone (we needed it for some hardware testing) and a leader balance (does it using partition reassignment without actually moving partitions). We have some scripts that the other SREs h

Re: Replication tools to move topics/partitions gradually

2015-05-24 Thread Henry Cai
Todd, This is very promising. Do you know when will we be able to see your tools released to public? On Sun, May 24, 2015 at 7:54 PM, Todd Palino wrote: > We've built tools on top of it that both build the list based on less > information (like "clone this broker to that one") and break it dow

Re: Replication tools to move topics/partitions gradually

2015-05-24 Thread Todd Palino
We've built tools on top of it that both build the list based on less information (like "clone this broker to that one") and break it down into a configurable number of discrete moves so it doesn't tank the cluster. And yes, I've finally started the process of departing them from the LinkedIn-s

Replication tools to move topics/partitions gradually

2015-05-24 Thread Henry Cai
We have a kafka cluster with 10 brokers and we are using the kafka replication tool (kafka-reassign-partitions.sh) when we need to add more brokers to the cluster. But this tool tends to move too many topic/partitions around at the same time which causes instability. Do we have an option to do it