So, it means that I should create topics with at least replication-factor=2 inspite of how many servers in a kafka cluster. If any server goes down or slows down then zookeeper will not go out-of-sync. Currently, my all topics are with eplication-factor= 1 and I got an issue that Zookeeper goes out of sync. So, increasing replication-factor will solve the issue?
Hafsa 2016-06-01 12:57 GMT+02:00 Ben Stopford <b...@confluent.io>: > Hi Hafa > > If you create a topic with replication-factor = 2, you can lose one of > them without losing data, so long as they were "in sync". Replicas can fall > out of sync if one of the machines runs slow. The system tracks in sync > replicas. These are exposed by JMX too. Check out the docs on replication > for more details: > > http://kafka.apache.org/090/documentation.html#replication < > http://kafka.apache.org/090/documentation.html#replication> > > B > > > On 1 Jun 2016, at 10:45, Hafsa Asif <hafsa.a...@matchinguu.com> wrote: > > > > Hello Jayesh, > > > > Thank you very much for such a good description. My further questions are > > (just to be my self clear about the concept). > > > > 1. If I have only one partition in a 'Topic' in a Kafka with following > > configuration, > > bin/kafka-topics.sh --create --zookeeper localhost:2181 > > --replication-factor 1 --partitions 1 --topic mytopic1 > > Then still I need to rebalance topic partitions while node > adding/removing > > in Kafka cluster? > > > > 2. What is the actual meaning of this line 'if all your topics have > atleast > > 2 insync replicas'. My understanding is that, I need to create replica of > > each topic in each server. e.g: I have two servers in a Kafka cluster > then > > I need to create topic 'mytopic1' in both servers. It helps to get rid of > > any problem while removing any of the server. > > > > I will look in detail into your provided link. Many thanks for this. > > > > Looking forward for the answers from also other Kafka ninjas as well :) > > > > Best, > > Hafsa > > > > 2016-05-31 18:50 GMT+02:00 Thakrar, Jayesh <jthak...@conversantmedia.com > >: > > > >> Hafsa, Florin > >> > >> First thing first, it is possible to scale a Kafka cluster up or down > >> (i.e. add/remove servers). > >> And as has been noted in this thread, after you add a server to a > cluster, > >> you need to rebalance the topic partitions in order to put the newly > added > >> server into use. > >> And similarly, before you remove a server, it is advised that you drain > >> off the data from the server to be removed (its not a hard requirement, > if > >> all your topics have atleast 2 insync replicas, including the server > being > >> removed and you intend to rebalance after server removal). > >> > >> However, "automating" the rebalancing of topic partitions is not > trivial. > >> > >> There is a KIP out there to help with the rebalancing , but lacks > details > >> - > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing > >> My guess is due to its non-trivial nature AND the number of cases one > >> needs to take care of - e.g. scaling up by 5% v/s scaling up by 50% in > say, > >> a 20 node cluster. > >> Furthermore, to be really effective, one needs to be cognizant of the > >> partition sizes, and with rack-awareness, the task becomes even more > >> involved. > >> > >> Regards, > >> Jayesh > >> > >> -----Original Message----- > >> From: Spico Florin [mailto:spicoflo...@gmail.com] > >> Sent: Tuesday, May 31, 2016 9:44 AM > >> To: users@kafka.apache.org > >> Subject: Re: Rebalancing issue while Kafka scaling > >> > >> Hi! > >> What version of Kafka you are using? What do you mean by "Kafka needs > >> rebalacing?" Rebalancing of what? Can you please be more specific. > >> > >> Regards, > >> Florin > >> > >> > >> > >> On Tue, May 31, 2016 at 4:58 PM, Hafsa Asif <hafsa.a...@matchinguu.com> > >> wrote: > >> > >>> Hello Folks, > >>> > >>> Today , my team members shows concern that whenever we increase node > >>> in Kafka cluster, Kafka needs rebalancing. The rebalancing is sort of > >>> manual and not-good step whenever scaling happens. Second, if Kafka > >>> scales up then it cannot be scale down. Please provide us proper > >>> guidance over this issue, may be we have not enough configuration > >> properties. > >>> > >>> Hafsa > >>> > >> > >> > >> > >> > >> This email and any files included with it may contain privileged, > >> proprietary and/or confidential information that is for the sole use > >> of the intended recipient(s). Any disclosure, copying, distribution, > >> posting, or use of the information contained in or attached to this > >> email is prohibited unless permitted by the sender. If you have > >> received this email in error, please immediately notify the sender > >> via return email, telephone, or fax and destroy this original > transmission > >> and its included files without reading or saving it in any manner. > >> Thank you. > >> > >
