Hi Hafsa, Not sure by what you mean "most tolerant cluster". If you mean that you want the cluster to be able to tolerate 9 of 10 servers to be down, then yes. But I would question - is your traffic activity, system load and storage requirement so low that it can be served by a single server? If so, then having a 10-server cluster is an over-kill. Note that the more replicas a partition has higher is the network traffic and load on the leader partition broker (server) as 1) all producers send data to that broker for that partition 2) all consumers read data from that broker for that partition 3) and all non-leader replica partition brokers replicate data from that partition. So determining replica is balance and "traditionally" both in HDFS and Kafka, 3 seems to be magic number.
As for the number of partitions (not replicas), its good to have as a multiple of the number of brokers/servers - but again there are several things to consider - e.g. storage requirements, expected load (network, I/O, etc.) across each topic and across the cluster as a whole. One rule of thumb (again a magic number) that I heard is that its good to have the number of partitions that result in total storage for the partition to be around 20-50 GB. There are a couple of good articles out there on Kafka cluster design - e.g. http://morebigdata.blogspot.com/2015/10/tips-for-successful-kafka-deployments.html Hope that helps. Jayesh -----Original Message----- From: Hafsa Asif [mailto:hafsa.a...@matchinguu.com] Sent: Wednesday, June 01, 2016 7:05 AM To: users@kafka.apache.org Cc: Spico Florin <spicoflo...@gmail.com> Subject: Re: Rebalancing issue while Kafka scaling Just for more info: If I have 10 servers in a cluster, so for the most tolerant cluster, do we need replication-factor = 10? That is also the issue for rebalancing the scaling of kafka cluster, that when we need to add server in a cluster then we also need to increase partitions in topics as well ? Best, Hafsa 2016-06-01 13:55 GMT+02:00 Ben Stopford <b...@confluent.io>: > Pretty much. It’s not actually related to zookeeper. > > Generalising a bit, replication factor 2 means Kafka can lose 1 > machine and be ok. > > B > > On 1 Jun 2016, at 12:46, Hafsa Asif <hafsa.a...@matchinguu.com> wrote: > > > > So, it means that I should create topics with at least > replication-factor=2 > > inspite of how many servers in a kafka cluster. If any server goes > > down > or > > slows down then zookeeper will not go out-of-sync. > > Currently, my all topics are with eplication-factor= 1 and I got an > > issue that Zookeeper goes out of sync. So, increasing > > replication-factor will solve the issue? > > > > Hafsa > > > > 2016-06-01 12:57 GMT+02:00 Ben Stopford <b...@confluent.io>: > > > >> Hi Hafa > >> > >> If you create a topic with replication-factor = 2, you can lose one > >> of them without losing data, so long as they were "in sync". > >> Replicas can > fall > >> out of sync if one of the machines runs slow. The system tracks in > >> sync replicas. These are exposed by JMX too. Check out the docs on > replication > >> for more details: > >> > >> http://kafka.apache.org/090/documentation.html#replication < > >> http://kafka.apache.org/090/documentation.html#replication> > >> > >> B > >> > >>> On 1 Jun 2016, at 10:45, Hafsa Asif <hafsa.a...@matchinguu.com> wrote: > >>> > >>> Hello Jayesh, > >>> > >>> Thank you very much for such a good description. My further > >>> questions > are > >>> (just to be my self clear about the concept). > >>> > >>> 1. If I have only one partition in a 'Topic' in a Kafka with > >>> following configuration, bin/kafka-topics.sh --create --zookeeper > >>> localhost:2181 --replication-factor 1 --partitions 1 --topic > >>> mytopic1 Then still I need to rebalance topic partitions while > >>> node > >> adding/removing > >>> in Kafka cluster? > >>> > >>> 2. What is the actual meaning of this line 'if all your topics > >>> have > >> atleast > >>> 2 insync replicas'. My understanding is that, I need to create > >>> replica > of > >>> each topic in each server. e.g: I have two servers in a Kafka > >>> cluster > >> then > >>> I need to create topic 'mytopic1' in both servers. It helps to get > >>> rid > of > >>> any problem while removing any of the server. > >>> > >>> I will look in detail into your provided link. Many thanks for this. > >>> > >>> Looking forward for the answers from also other Kafka ninjas as > >>> well :) > >>> > >>> Best, > >>> Hafsa > >>> > >>> 2016-05-31 18:50 GMT+02:00 Thakrar, Jayesh < > jthak...@conversantmedia.com > >>> : > >>> > >>>> Hafsa, Florin > >>>> > >>>> First thing first, it is possible to scale a Kafka cluster up or > >>>> down (i.e. add/remove servers). > >>>> And as has been noted in this thread, after you add a server to a > >> cluster, > >>>> you need to rebalance the topic partitions in order to put the > >>>> newly > >> added > >>>> server into use. > >>>> And similarly, before you remove a server, it is advised that you > drain > >>>> off the data from the server to be removed (its not a hard > requirement, > >> if > >>>> all your topics have atleast 2 insync replicas, including the > >>>> server > >> being > >>>> removed and you intend to rebalance after server removal). > >>>> > >>>> However, "automating" the rebalancing of topic partitions is not > >> trivial. > >>>> > >>>> There is a KIP out there to help with the rebalancing , but lacks > >> details > >>>> - > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassign > ment+partition+logic+for+rebalancing > >>>> My guess is due to its non-trivial nature AND the number of cases > >>>> one needs to take care of - e.g. scaling up by 5% v/s scaling up > >>>> by 50% in > >> say, > >>>> a 20 node cluster. > >>>> Furthermore, to be really effective, one needs to be cognizant of > >>>> the partition sizes, and with rack-awareness, the task becomes > >>>> even more involved. > >>>> > >>>> Regards, > >>>> Jayesh > >>>> > >>>> -----Original Message----- > >>>> From: Spico Florin [mailto:spicoflo...@gmail.com] > >>>> Sent: Tuesday, May 31, 2016 9:44 AM > >>>> To: users@kafka.apache.org > >>>> Subject: Re: Rebalancing issue while Kafka scaling > >>>> > >>>> Hi! > >>>> What version of Kafka you are using? What do you mean by "Kafka > >>>> needs rebalacing?" Rebalancing of what? Can you please be more specific. > >>>> > >>>> Regards, > >>>> Florin > >>>> > >>>> > >>>> > >>>> On Tue, May 31, 2016 at 4:58 PM, Hafsa Asif < > hafsa.a...@matchinguu.com> > >>>> wrote: > >>>> > >>>>> Hello Folks, > >>>>> > >>>>> Today , my team members shows concern that whenever we increase > >>>>> node in Kafka cluster, Kafka needs rebalancing. The rebalancing > >>>>> is sort of manual and not-good step whenever scaling happens. > >>>>> Second, if Kafka scales up then it cannot be scale down. Please > >>>>> provide us proper guidance over this issue, may be we have not > >>>>> enough configuration > >>>> properties. > >>>>> > >>>>> Hafsa > >>>>> > >>>> > >>>> > >>>> > >>>> > >>>> This email and any files included with it may contain privileged, > >>>> proprietary and/or confidential information that is for the sole > >>>> use of the intended recipient(s). Any disclosure, copying, > >>>> distribution, posting, or use of the information contained in or > >>>> attached to this email is prohibited unless permitted by the > >>>> sender. If you have received this email in error, please > >>>> immediately notify the sender via return email, telephone, or fax > >>>> and destroy this original > >> transmission > >>>> and its included files without reading or saving it in any manner. > >>>> Thank you. > >>>> > >> > >> > > This email and any files included with it may contain privileged, proprietary and/or confidential information that is for the sole use of the intended recipient(s). Any disclosure, copying, distribution, posting, or use of the information contained in or attached to this email is prohibited unless permitted by the sender. If you have received this email in error, please immediately notify the sender via return email, telephone, or fax and destroy this original transmission and its included files without reading or saving it in any manner. Thank you.
