Hi, Please help/guide me to identify the size of the cluster.
Thanks and Regards, Gowtham S. On Fri, 24 Jan 2020 at 14:31, Gowtham S <gowtham.co....@gmail.com> wrote: > Hi, > > We are in the process of deploying Kafka in our service. We need to decide > the machine capacity plan, we arrived at the below formulae for deriving > total machine capacity. > > Total Broker machine size = Message size per second * Retention period * > Replication Factor > > Am I need to consider the topic, index files in the calculation? Please > help/guide me if i missing any param required in the formulae. > > Index file calculation (Reference > <https://issues.apache.org/jira/browse/KAFKA-3300>) > > Currently, the initial/max size of offset index file is configured by > log.index.max.bytes. > This will be the offset index file size for the active log segment until it > rolls out. > > Theoretically, we can calculate the upper bound of offset index size using > the following formula: > > log.segment.bytes / index.interval.bytes * 8 > > With the default setting the bytes needed for an offset index size is 1GB > / 4K * 8 = 2MB. And the default log.index.max.bytes is 10MB. > > > Retention Period = Retention period + (log.retention.check.interval.ms + > log.segment.delete.delay.ms ) / 1000 > = 86400 + (30000 + 6000)/1000 > = 86400 + 360 > = 86760 Seconds > > Total Broker machine size = Message size per second * Retention period * > Replication Factor > = 90 MB/Sec * 86760 * 3 > = 23425200 MB > = 23.4252 Tb > > *Machine Configuration* > 6 Brokers with 3 ZK > > *Kafka (per machine)* > Disk Space - 2 * 2TB > RAM - 128 GB > CPU - 40 core > > Thanks and Regards > Gowtham.S >
