Ah, got it! There's no easy way to transfer leadership on command, but you could use the reassignment tool to change the preferred leader (and nothing else) and then trigger preferred leader election.
Gwen On Thu, Jan 14, 2016 at 11:30 AM, Luke Steensen < luke.steen...@braintreepayments.com> wrote: > Hi Gwen, > > 1. I sent a message to this list a couple days ago with the subject > "Controlled shutdown not relinquishing leadership of all partitions" > describing the issue I saw. Sorry there's not a lot of detail on the > controlled shutdown part, but I've had trouble reproducing outside of our > specific deployment. > > 2. Yes, that makes sense. Sorry, I was implicitly assuming tight timeouts > and at least one retry. > > 3. Right, my understanding is that it doesn't change the preferred leader, > it just triggers a more graceful leader election than would occur if the > broker were killed unexpectedly. I was basically asking if there's a way to > move leadership away from a broker independently of shutting it down. That > would really just be a workaround for the controlled shutdown issues we > experienced. > > 4. Yep, we rely on exactly this behavior when replacing nodes. It's very > helpful :) > > Thanks! > Luke > > > On Thu, Jan 14, 2016 at 10:07 AM, Gwen Shapira <g...@confluent.io> wrote: > > > Hi, > > > > 1. If you had problems with controlled shutdown, we need to know. Maybe > > open a thread to discuss? > > 2. Controlled shutdown is only used to reduce the downtime involved in > > large number of leader elections. New leaders will get elected in any > case. > > 3. Controlled (or uncontrolled shutdown) does not change the preferred > > leader. This happens only on re-assignment. > > 4. #3 relies on the fact that if you are a brand new broker with > absolutely > > no data joining the cluster with id = "n", and the replica-map shows that > > broker "n" has certain partitions (because we never assigned them away), > > the new broker will immediately become follower for these partitions and > > start replicating the missing data. > > This makes automatic recover much easier. > > > > Gwen > > > > On Thu, Jan 14, 2016 at 10:49 AM, Luke Steensen < > > luke.steen...@braintreepayments.com> wrote: > > > > > Hello, > > > > > > For #3, I assume this relies on controlled shutdown to transfer > > leadership > > > gracefully? Or is there some way to use partition reassignment to set > the > > > preferred leader of each partition? I ask because we've run into some > > > problems relying on controlled shutdown and having a separate > verifiable > > > step would be nice. > > > > > > Thanks, > > > Luke > > > > > > > > > On Thu, Jan 14, 2016 at 9:36 AM, Gwen Shapira <g...@confluent.io> > wrote: > > > > > > > Hi, > > > > > > > > There was a Jira to add "remove broker" option to the > > > > partition-reassignment tool. I think it died in a long discussion > > trying > > > to > > > > solve a harder problem... > > > > > > > > To your work-around - it is an acceptable work-around. > > > > > > > > Few improvements: > > > > 1. Manually edit the resulting assignment json to avoid unnecessary > > > moves. > > > > Or even create your own assignment (either manually or using a small > > > > script). > > > > 2. We don't throttle the partition move automatically, so it can > easily > > > > take over the network if you are not careful. Therefore running the > > > > reassignment tools multiple times to move partitions one-by-one is > > often > > > > safer. > > > > 3. If you don't mean to permanently reduce the number of brokers but > > > rather > > > > to replace a broker, don't reassign. Just take down the existing > broker > > > and > > > > give the new one the same ID. > > > > > > > > Hope this helps, > > > > > > > > Gwen > > > > > > > > On Wed, Jan 6, 2016 at 10:14 AM, Tom Crayford <tcrayf...@heroku.com> > > > > wrote: > > > > > > > > > Hi there, > > > > > > > > > > Kafka's `kafka-reassign-partitions.sh` tool currently has no > > mechanism > > > > for > > > > > removing brokers. However, it does have the ability to generate > > > partition > > > > > plans across arbitrary sets of brokers, by using `--generate`, > > passing > > > > all > > > > > the topics in the cluster into it, then passing the generated plan > to > > > > > --execute. > > > > > > > > > > This isn't ideal, because it (from my understanding), potentially > > moves > > > > all > > > > > the partitions in the entire cluster around, but it should work > fine, > > > and > > > > > stop Kafka from having the partitions assigned to a broker that no > > > longer > > > > > exists. > > > > > > > > > > Am I missing something there? Or is this a reasonable workaround > > until > > > > > better partition reassignment tools turn up in the future? > > > > > > > > > > Thanks > > > > > > > > > > Tom > > > > > > > > > > > > > > >
