For fault -tolerance you want the replicated partitions to be on different physical servers. I think a better set is to have 3 brokers, 9 partitions and a replication factor of 2. This will ensure that the replicated data is always on a different physical hosts.
-Dave Dave Tauzell | Senior Software Engineer | Surescripts O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com Connect with us: Twitter I LinkedIn I Facebook I YouTube -----Original Message----- From: Amit K [mailto:amitk....@gmail.com] Sent: Wednesday, July 20, 2016 11:40 AM To: users@kafka.apache.org Subject: Re: Regarding kafka partition and replication Thanks for reply. The infrastructure, which I mentioned, is already in place and is getting used so can not alter it. But given that, if I have 9 partitions with replication factor of 2, will that help me to have a good fault tolerant and optimal, in regard with hardware use, system? Thanks, Amit On Tue, Jul 19, 2016 at 6:48 PM, Tauzell, Dave <dave.tauz...@surescripts.com > wrote: > Having multiple brokers on the same node has a couple of problems for > a production installation: > > 1. You'll have multiple brokers contending for disk and memory > resources 2. You could have your partitions replicated to the same > node which means if that node fails you would lose data. > > I think you are better off having 3 nodes with 3 brokers. You can keep > with 9 partitions in case you want to add physical nodes in the future > and have a replication factor of 2 or 3. > > -Dave > > Dave Tauzell | Senior Software Engineer | Surescripts > O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com > Connect with us: Twitter I LinkedIn I Facebook I YouTube > > > -----Original Message----- > From: Amit K [mailto:amitk....@gmail.com] > Sent: Monday, July 18, 2016 8:55 PM > To: users@kafka.apache.org > Subject: Regarding kafka partition and replication > > Hi, > > I have 3 nodes, each with 3 brokers, Kafka cluster along with 3 > zookeeper cluster. So total 9 brokers spread across 3 different > machines. I am adhered to Kafka 0.9. > > In order to optimally use the infrastructure for 2 topics (as of now, > is not expected to grow drastically in near future), I am thinking of > having 9 partitions with 3 (or 6?) replication factor. Will this help > me having good distribution of partitions and replicas across brokers? > System does not have hugh load (<50 requests/sec of less than 1 kb > load each) as of now and is neither expected to get higher load than this. > > If this replication and partition does not help, please suggest a > better topic partition and replication strategy. > > Also please guide me with any articles or documents about setting up > multi node Kafka cluster in regard to partition, replication, general > properties to be used (a kind of good practice etc.) > > Thanks, > Amit > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of > the individual or entity to whom they are addressed. If you have > received this e-mail in error, please notify the sender by reply > e-mail immediately and destroy all copies of the e-mail and any attachments. > This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
