Hi Naveen, a very broad question, but to use Kafka as a backbone of your infrastructure, the brokers need to be on machines with enough disk to store the expected data, and with good network interface capacity - we use 10Gbps NICs.
Sizing disks is a case of knowing how long you want retain data for, how many replicas you want to have, and how much data you expect, and how you'll partition it. For example, if your topic will have 1TB of data produced onto it in a day, and you want to have 3 replicas, retaining data for 14 days, you'll need 3 * 14 * 1TB disk space across your cluster to fit it. But you also need to ensure that one partition can fit on disk on a broker. So if the above topic has 20 partitions, you'll need to make sure that each broker has a disk that can fit 50GB * 14. Plus you need some additional space for Kafka metadata and OS overhead etc. Hope that helps, Liam Clarke On Thu, 20 Feb. 2020, 7:02 pm Naveen Kumar M, <eai.naveenkuma...@gmail.com> wrote: > Hello Kafka Community, > > Kindly help me on this. > > Thanks and regards, > Naveen > > On Wed, Feb 19, 2020, 1:41 PM Naveen Kumar M <eai.naveenkuma...@gmail.com> > wrote: > > > Hello Team, > > > > Kindly help me to understand hardware assessment details to setup Kafka > as > > messaging broker to de-couple applications/systems. > > > > Thanks and regards, > > Naveen > > >
