Thanks for all your pieces of advice!
I am working on my script to improve it.
I am trying to find a way to select the best partition to relocate it to
the newly added broker (taking in account the number of partition leaded by
the replica brokers).
I really want to avoid swaps to rebalance the leadership at the end of my
script which will trigger replica moves.
Just a quick question about a partition move:
Supposed that I have the following the partitionnement plan:
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2,3]}
}
and I submit this Json file to move partitions
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[1,3,2]} }
Is Kafka gonna switch replicas between broker 2 and 3?
Thanks,
On Fri, Jul 11, 2014 at 2:56 PM, Clark Haskins <
chask...@linkedin.com.invalid> wrote:
> I have written such a script. It balances the cluster by the data size on
> disk. It is written using lots of internal tools which is why its not
> open-sourced. I plan to re-write it without the internal tooling.
>
> In terms of leader balancing, when using the partition-reassignemnt
> script, whichever broker is specified first within the list of brokers
> will be the leader.
>
> -Clark
>
>
> Clark Elliott Haskins III
> LinkedIn DDS Site Reliability Engineer
> Kafka, Zookeeper, Samza SRE
> Mobile: 505.385.1484
> BlueJeans: https://www.bluejeans.com/chaskins
>
>
> chask...@linkedin.com
> https://www.linkedin.com/in/clarkhaskins
> There is no place like 127.0.0.1
>
>
>
>
> On 7/10/14, 6:07 PM, "Florian Dambrine" <flor...@gumgum.com> wrote:
>
> >Thanks for your answer,
> >
> >Indeed, I have already worked on this kind of script. I ended up with 800
> >lines of groovy script that rebalance partitions across the cluster and
> >minimizing the number of partition moves. I also worked on the partition
> >leadership balancing.
> >
> >I still have to work on my script because I end up with some unbalanced
> >leaders. The number of partition leaded by one node is on average 17 but I
> >have one node that ends with 12 leads and an other with 21.
> >I am gonna introduce swaps to re equilibrate the leadership.
> >
> >Have you ever worked on this kind of script? I could not find any
> >open-source code on GitHub...
> >
> >Do you have any suggestions?
> >
> >Just in case if you want to have a look I have published my code on
> >GitHub (
> >
> https://github.com/Lowess/Kafka/blob/master/com/gumgum/kafka/KafkaManualPa
> >rtitionRebalancer.groovy
> >)
> >
> >Thanks
>
>
--
*Florian Dambrine* | Intern, Big Data
*GumGum* <http://www.gumgum.com/> | *Ads that stick*
209-797-3994 | flor...@gumgum.com
