Hello, For #3, I assume this relies on controlled shutdown to transfer leadership gracefully? Or is there some way to use partition reassignment to set the preferred leader of each partition? I ask because we've run into some problems relying on controlled shutdown and having a separate verifiable step would be nice.
Thanks, Luke On Thu, Jan 14, 2016 at 9:36 AM, Gwen Shapira <g...@confluent.io> wrote: > Hi, > > There was a Jira to add "remove broker" option to the > partition-reassignment tool. I think it died in a long discussion trying to > solve a harder problem... > > To your work-around - it is an acceptable work-around. > > Few improvements: > 1. Manually edit the resulting assignment json to avoid unnecessary moves. > Or even create your own assignment (either manually or using a small > script). > 2. We don't throttle the partition move automatically, so it can easily > take over the network if you are not careful. Therefore running the > reassignment tools multiple times to move partitions one-by-one is often > safer. > 3. If you don't mean to permanently reduce the number of brokers but rather > to replace a broker, don't reassign. Just take down the existing broker and > give the new one the same ID. > > Hope this helps, > > Gwen > > On Wed, Jan 6, 2016 at 10:14 AM, Tom Crayford <tcrayf...@heroku.com> > wrote: > > > Hi there, > > > > Kafka's `kafka-reassign-partitions.sh` tool currently has no mechanism > for > > removing brokers. However, it does have the ability to generate partition > > plans across arbitrary sets of brokers, by using `--generate`, passing > all > > the topics in the cluster into it, then passing the generated plan to > > --execute. > > > > This isn't ideal, because it (from my understanding), potentially moves > all > > the partitions in the entire cluster around, but it should work fine, and > > stop Kafka from having the partitions assigned to a broker that no longer > > exists. > > > > Am I missing something there? Or is this a reasonable workaround until > > better partition reassignment tools turn up in the future? > > > > Thanks > > > > Tom > > >
