Hi, I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like to add a new broker in the cluster. Each kafka instance has 400Go of data and we are using replication factor equals to 3 with 50 partitions for each topics.
I have check the documentation and especially the section dealing with the cluster expansion ( https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion). So far, I have tried the automatic way to reassign partitions using the --topics-to-move-json-file option on 1 small topic. Here is the command that I have ran: content of topic-to-move.json: { "topics": [ {"topic": "SMALL_TOPIC"}], "version":1 } command: ./bin/kafka-reassign-partitions.sh --zookeeper zkDns:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5,6,7,8,9" --generate Has you can see I specified a broker list including all the broker from the cluster (including the newly added kafka node number 9). First of all is it the right way to do that? As I am using replication factor = 3, I have to specify at least 3 brokers. I thought that specifying the entire list of broker will keep the cluster balanced (and it did). Here is my main question, I looked the Ganglia ui in order to see the impact of this command and here is the networking graph ( http://s18.postimg.org/hokyoeyax/kafka_network.png) As you can see on the screen-shot the network traffic goes pretty high. That is why I am afraid to run this tool on bigger topics. I also took a look into the manual way of reassigning partitions. the problem is that I always have to specify 3 replicas even if I just want to move only one to my new broker. Do you know any tool or script to help me in this process? Thanks for your help, Florian -- *Florian Dambrine* | Intern, Big Data *GumGum* <http://www.gumgum.com/> | *Ads that stick* 209-797-3994 | flor...@gumgum.com
