Hi Hafa If you create a topic with replication-factor = 2, you can lose one of them without losing data, so long as they were "in sync". Replicas can fall out of sync if one of the machines runs slow. The system tracks in sync replicas. These are exposed by JMX too. Check out the docs on replication for more details:
http://kafka.apache.org/090/documentation.html#replication <http://kafka.apache.org/090/documentation.html#replication> B > On 1 Jun 2016, at 10:45, Hafsa Asif <hafsa.a...@matchinguu.com> wrote: > > Hello Jayesh, > > Thank you very much for such a good description. My further questions are > (just to be my self clear about the concept). > > 1. If I have only one partition in a 'Topic' in a Kafka with following > configuration, > bin/kafka-topics.sh --create --zookeeper localhost:2181 > --replication-factor 1 --partitions 1 --topic mytopic1 > Then still I need to rebalance topic partitions while node adding/removing > in Kafka cluster? > > 2. What is the actual meaning of this line 'if all your topics have atleast > 2 insync replicas'. My understanding is that, I need to create replica of > each topic in each server. e.g: I have two servers in a Kafka cluster then > I need to create topic 'mytopic1' in both servers. It helps to get rid of > any problem while removing any of the server. > > I will look in detail into your provided link. Many thanks for this. > > Looking forward for the answers from also other Kafka ninjas as well :) > > Best, > Hafsa > > 2016-05-31 18:50 GMT+02:00 Thakrar, Jayesh <jthak...@conversantmedia.com>: > >> Hafsa, Florin >> >> First thing first, it is possible to scale a Kafka cluster up or down >> (i.e. add/remove servers). >> And as has been noted in this thread, after you add a server to a cluster, >> you need to rebalance the topic partitions in order to put the newly added >> server into use. >> And similarly, before you remove a server, it is advised that you drain >> off the data from the server to be removed (its not a hard requirement, if >> all your topics have atleast 2 insync replicas, including the server being >> removed and you intend to rebalance after server removal). >> >> However, "automating" the rebalancing of topic partitions is not trivial. >> >> There is a KIP out there to help with the rebalancing , but lacks details >> - >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing >> My guess is due to its non-trivial nature AND the number of cases one >> needs to take care of - e.g. scaling up by 5% v/s scaling up by 50% in say, >> a 20 node cluster. >> Furthermore, to be really effective, one needs to be cognizant of the >> partition sizes, and with rack-awareness, the task becomes even more >> involved. >> >> Regards, >> Jayesh >> >> -----Original Message----- >> From: Spico Florin [mailto:spicoflo...@gmail.com] >> Sent: Tuesday, May 31, 2016 9:44 AM >> To: users@kafka.apache.org >> Subject: Re: Rebalancing issue while Kafka scaling >> >> Hi! >> What version of Kafka you are using? What do you mean by "Kafka needs >> rebalacing?" Rebalancing of what? Can you please be more specific. >> >> Regards, >> Florin >> >> >> >> On Tue, May 31, 2016 at 4:58 PM, Hafsa Asif <hafsa.a...@matchinguu.com> >> wrote: >> >>> Hello Folks, >>> >>> Today , my team members shows concern that whenever we increase node >>> in Kafka cluster, Kafka needs rebalancing. The rebalancing is sort of >>> manual and not-good step whenever scaling happens. Second, if Kafka >>> scales up then it cannot be scale down. Please provide us proper >>> guidance over this issue, may be we have not enough configuration >> properties. >>> >>> Hafsa >>> >> >> >> >> >> This email and any files included with it may contain privileged, >> proprietary and/or confidential information that is for the sole use >> of the intended recipient(s). Any disclosure, copying, distribution, >> posting, or use of the information contained in or attached to this >> email is prohibited unless permitted by the sender. If you have >> received this email in error, please immediately notify the sender >> via return email, telephone, or fax and destroy this original transmission >> and its included files without reading or saving it in any manner. >> Thank you. >>
