Re: Partition rebalancing after broker removal

2016-01-14 Thread Luke Steensen
No worries, glad to have the functionality! Thanks for your help. Luke On Thu, Jan 14, 2016 at 10:58 AM, Gwen Shapira wrote: > Yep. That tool is not our best documented :( > > On Thu, Jan 14, 2016 at 11:49 AM, Luke Steensen < > luke.steen...@braintreepayments.com> wrote: > > > Is the preferred

Re: Partition rebalancing after broker removal

2016-01-14 Thread Gwen Shapira
Yep. That tool is not our best documented :( On Thu, Jan 14, 2016 at 11:49 AM, Luke Steensen < luke.steen...@braintreepayments.com> wrote: > Is the preferred leader the first replica in the list passed to the > reassignment tool? I don't see it specifically called out in the json file > format. >

Re: Partition rebalancing after broker removal

2016-01-14 Thread Luke Steensen
Is the preferred leader the first replica in the list passed to the reassignment tool? I don't see it specifically called out in the json file format. On Thu, Jan 14, 2016 at 10:42 AM, Gwen Shapira wrote: > Ah, got it! > > There's no easy way to transfer leadership on command, but you could use

Re: Partition rebalancing after broker removal

2016-01-14 Thread Gwen Shapira
Ah, got it! There's no easy way to transfer leadership on command, but you could use the reassignment tool to change the preferred leader (and nothing else) and then trigger preferred leader election. Gwen On Thu, Jan 14, 2016 at 11:30 AM, Luke Steensen < luke.steen...@braintreepayments.com> wro

Re: Partition rebalancing after broker removal

2016-01-14 Thread Luke Steensen
Hi Gwen, 1. I sent a message to this list a couple days ago with the subject "Controlled shutdown not relinquishing leadership of all partitions" describing the issue I saw. Sorry there's not a lot of detail on the controlled shutdown part, but I've had trouble reproducing outside of our specific

Re: Partition rebalancing after broker removal

2016-01-14 Thread Gwen Shapira
Hi, 1. If you had problems with controlled shutdown, we need to know. Maybe open a thread to discuss? 2. Controlled shutdown is only used to reduce the downtime involved in large number of leader elections. New leaders will get elected in any case. 3. Controlled (or uncontrolled shutdown) does not

Re: Partition rebalancing after broker removal

2016-01-14 Thread Luke Steensen
Hello, For #3, I assume this relies on controlled shutdown to transfer leadership gracefully? Or is there some way to use partition reassignment to set the preferred leader of each partition? I ask because we've run into some problems relying on controlled shutdown and having a separate verifiable

Re: Partition rebalancing after broker removal

2016-01-14 Thread Gwen Shapira
Hi, There was a Jira to add "remove broker" option to the partition-reassignment tool. I think it died in a long discussion trying to solve a harder problem... To your work-around - it is an acceptable work-around. Few improvements: 1. Manually edit the resulting assignment json to avoid unneces

Partition rebalancing after broker removal

2016-01-06 Thread Tom Crayford
Hi there, Kafka's `kafka-reassign-partitions.sh` tool currently has no mechanism for removing brokers. However, it does have the ability to generate partition plans across arbitrary sets of brokers, by using `--generate`, passing all the topics in the cluster into it, then passing the generated pl