Yep. That tool is not our best documented :( On Thu, Jan 14, 2016 at 11:49 AM, Luke Steensen < luke.steen...@braintreepayments.com> wrote:
> Is the preferred leader the first replica in the list passed to the > reassignment tool? I don't see it specifically called out in the json file > format. > > > On Thu, Jan 14, 2016 at 10:42 AM, Gwen Shapira <g...@confluent.io> wrote: > > > Ah, got it! > > > > There's no easy way to transfer leadership on command, but you could use > > the reassignment tool to change the preferred leader (and nothing else) > and > > then trigger preferred leader election. > > > > Gwen > > > > On Thu, Jan 14, 2016 at 11:30 AM, Luke Steensen < > > luke.steen...@braintreepayments.com> wrote: > > > > > Hi Gwen, > > > > > > 1. I sent a message to this list a couple days ago with the subject > > > "Controlled shutdown not relinquishing leadership of all partitions" > > > describing the issue I saw. Sorry there's not a lot of detail on the > > > controlled shutdown part, but I've had trouble reproducing outside of > our > > > specific deployment. > > > > > > 2. Yes, that makes sense. Sorry, I was implicitly assuming tight > timeouts > > > and at least one retry. > > > > > > 3. Right, my understanding is that it doesn't change the preferred > > leader, > > > it just triggers a more graceful leader election than would occur if > the > > > broker were killed unexpectedly. I was basically asking if there's a > way > > to > > > move leadership away from a broker independently of shutting it down. > > That > > > would really just be a workaround for the controlled shutdown issues we > > > experienced. > > > > > > 4. Yep, we rely on exactly this behavior when replacing nodes. It's > very > > > helpful :) > > > > > > Thanks! > > > Luke > > > > > > > > > On Thu, Jan 14, 2016 at 10:07 AM, Gwen Shapira <g...@confluent.io> > > wrote: > > > > > > > Hi, > > > > > > > > 1. If you had problems with controlled shutdown, we need to know. > Maybe > > > > open a thread to discuss? > > > > 2. Controlled shutdown is only used to reduce the downtime involved > in > > > > large number of leader elections. New leaders will get elected in any > > > case. > > > > 3. Controlled (or uncontrolled shutdown) does not change the > preferred > > > > leader. This happens only on re-assignment. > > > > 4. #3 relies on the fact that if you are a brand new broker with > > > absolutely > > > > no data joining the cluster with id = "n", and the replica-map shows > > that > > > > broker "n" has certain partitions (because we never assigned them > > away), > > > > the new broker will immediately become follower for these partitions > > and > > > > start replicating the missing data. > > > > This makes automatic recover much easier. > > > > > > > > Gwen > > > > > > > > On Thu, Jan 14, 2016 at 10:49 AM, Luke Steensen < > > > > luke.steen...@braintreepayments.com> wrote: > > > > > > > > > Hello, > > > > > > > > > > For #3, I assume this relies on controlled shutdown to transfer > > > > leadership > > > > > gracefully? Or is there some way to use partition reassignment to > set > > > the > > > > > preferred leader of each partition? I ask because we've run into > some > > > > > problems relying on controlled shutdown and having a separate > > > verifiable > > > > > step would be nice. > > > > > > > > > > Thanks, > > > > > Luke > > > > > > > > > > > > > > > On Thu, Jan 14, 2016 at 9:36 AM, Gwen Shapira <g...@confluent.io> > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > There was a Jira to add "remove broker" option to the > > > > > > partition-reassignment tool. I think it died in a long discussion > > > > trying > > > > > to > > > > > > solve a harder problem... > > > > > > > > > > > > To your work-around - it is an acceptable work-around. > > > > > > > > > > > > Few improvements: > > > > > > 1. Manually edit the resulting assignment json to avoid > unnecessary > > > > > moves. > > > > > > Or even create your own assignment (either manually or using a > > small > > > > > > script). > > > > > > 2. We don't throttle the partition move automatically, so it can > > > easily > > > > > > take over the network if you are not careful. Therefore running > the > > > > > > reassignment tools multiple times to move partitions one-by-one > is > > > > often > > > > > > safer. > > > > > > 3. If you don't mean to permanently reduce the number of brokers > > but > > > > > rather > > > > > > to replace a broker, don't reassign. Just take down the existing > > > broker > > > > > and > > > > > > give the new one the same ID. > > > > > > > > > > > > Hope this helps, > > > > > > > > > > > > Gwen > > > > > > > > > > > > On Wed, Jan 6, 2016 at 10:14 AM, Tom Crayford < > > tcrayf...@heroku.com> > > > > > > wrote: > > > > > > > > > > > > > Hi there, > > > > > > > > > > > > > > Kafka's `kafka-reassign-partitions.sh` tool currently has no > > > > mechanism > > > > > > for > > > > > > > removing brokers. However, it does have the ability to generate > > > > > partition > > > > > > > plans across arbitrary sets of brokers, by using `--generate`, > > > > passing > > > > > > all > > > > > > > the topics in the cluster into it, then passing the generated > > plan > > > to > > > > > > > --execute. > > > > > > > > > > > > > > This isn't ideal, because it (from my understanding), > potentially > > > > moves > > > > > > all > > > > > > > the partitions in the entire cluster around, but it should work > > > fine, > > > > > and > > > > > > > stop Kafka from having the partitions assigned to a broker that > > no > > > > > longer > > > > > > > exists. > > > > > > > > > > > > > > Am I missing something there? Or is this a reasonable > workaround > > > > until > > > > > > > better partition reassignment tools turn up in the future? > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > Tom > > > > > > > > > > > > > > > > > > > > > > > > > > > >
