Hafsa, Florin First thing first, it is possible to scale a Kafka cluster up or down (i.e. add/remove servers). And as has been noted in this thread, after you add a server to a cluster, you need to rebalance the topic partitions in order to put the newly added server into use. And similarly, before you remove a server, it is advised that you drain off the data from the server to be removed (its not a hard requirement, if all your topics have atleast 2 insync replicas, including the server being removed and you intend to rebalance after server removal).
However, "automating" the rebalancing of topic partitions is not trivial. There is a KIP out there to help with the rebalancing , but lacks details - https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing My guess is due to its non-trivial nature AND the number of cases one needs to take care of - e.g. scaling up by 5% v/s scaling up by 50% in say, a 20 node cluster. Furthermore, to be really effective, one needs to be cognizant of the partition sizes, and with rack-awareness, the task becomes even more involved. Regards, Jayesh -----Original Message----- From: Spico Florin [mailto:spicoflo...@gmail.com] Sent: Tuesday, May 31, 2016 9:44 AM To: users@kafka.apache.org Subject: Re: Rebalancing issue while Kafka scaling Hi! What version of Kafka you are using? What do you mean by "Kafka needs rebalacing?" Rebalancing of what? Can you please be more specific. Regards, Florin On Tue, May 31, 2016 at 4:58 PM, Hafsa Asif <hafsa.a...@matchinguu.com> wrote: > Hello Folks, > > Today , my team members shows concern that whenever we increase node > in Kafka cluster, Kafka needs rebalancing. The rebalancing is sort of > manual and not-good step whenever scaling happens. Second, if Kafka > scales up then it cannot be scale down. Please provide us proper > guidance over this issue, may be we have not enough configuration properties. > > Hafsa > This email and any files included with it may contain privileged, proprietary and/or confidential information that is for the sole use of the intended recipient(s). Any disclosure, copying, distribution, posting, or use of the information contained in or attached to this email is prohibited unless permitted by the sender. If you have received this email in error, please immediately notify the sender via return email, telephone, or fax and destroy this original transmission and its included files without reading or saving it in any manner. Thank you.
