Hello Jayesh, Thank you very much for such a good description. My further questions are (just to be my self clear about the concept).
1. If I have only one partition in a 'Topic' in a Kafka with following configuration, bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mytopic1 Then still I need to rebalance topic partitions while node adding/removing in Kafka cluster? 2. What is the actual meaning of this line 'if all your topics have atleast 2 insync replicas'. My understanding is that, I need to create replica of each topic in each server. e.g: I have two servers in a Kafka cluster then I need to create topic 'mytopic1' in both servers. It helps to get rid of any problem while removing any of the server. I will look in detail into your provided link. Many thanks for this. Looking forward for the answers from also other Kafka ninjas as well :) Best, Hafsa 2016-05-31 18:50 GMT+02:00 Thakrar, Jayesh <jthak...@conversantmedia.com>: > Hafsa, Florin > > First thing first, it is possible to scale a Kafka cluster up or down > (i.e. add/remove servers). > And as has been noted in this thread, after you add a server to a cluster, > you need to rebalance the topic partitions in order to put the newly added > server into use. > And similarly, before you remove a server, it is advised that you drain > off the data from the server to be removed (its not a hard requirement, if > all your topics have atleast 2 insync replicas, including the server being > removed and you intend to rebalance after server removal). > > However, "automating" the rebalancing of topic partitions is not trivial. > > There is a KIP out there to help with the rebalancing , but lacks details > - > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing > My guess is due to its non-trivial nature AND the number of cases one > needs to take care of - e.g. scaling up by 5% v/s scaling up by 50% in say, > a 20 node cluster. > Furthermore, to be really effective, one needs to be cognizant of the > partition sizes, and with rack-awareness, the task becomes even more > involved. > > Regards, > Jayesh > > -----Original Message----- > From: Spico Florin [mailto:spicoflo...@gmail.com] > Sent: Tuesday, May 31, 2016 9:44 AM > To: users@kafka.apache.org > Subject: Re: Rebalancing issue while Kafka scaling > > Hi! > What version of Kafka you are using? What do you mean by "Kafka needs > rebalacing?" Rebalancing of what? Can you please be more specific. > > Regards, > Florin > > > > On Tue, May 31, 2016 at 4:58 PM, Hafsa Asif <hafsa.a...@matchinguu.com> > wrote: > > > Hello Folks, > > > > Today , my team members shows concern that whenever we increase node > > in Kafka cluster, Kafka needs rebalancing. The rebalancing is sort of > > manual and not-good step whenever scaling happens. Second, if Kafka > > scales up then it cannot be scale down. Please provide us proper > > guidance over this issue, may be we have not enough configuration > properties. > > > > Hafsa > > > > > > > This email and any files included with it may contain privileged, > proprietary and/or confidential information that is for the sole use > of the intended recipient(s). Any disclosure, copying, distribution, > posting, or use of the information contained in or attached to this > email is prohibited unless permitted by the sender. If you have > received this email in error, please immediately notify the sender > via return email, telephone, or fax and destroy this original transmission > and its included files without reading or saving it in any manner. > Thank you. >
