Hello Gowtham, You need to include the size of offset and time index files in your calculations, plus potentially transaction indexes. If you use default values, that means 10 MB each for every log segment, which default size is itself 1 GB.
Alexandre Le lun. 27 janv. 2020 à 08:51, Gowtham S <gowtham.co....@gmail.com> a écrit : > > Hi, > > Please help/guide me to identify the size of the cluster. > > Thanks and Regards, > Gowtham S. > > > On Fri, 24 Jan 2020 at 14:31, Gowtham S <gowtham.co....@gmail.com> wrote: > > > Hi, > > > > We are in the process of deploying Kafka in our service. We need to decide > > the machine capacity plan, we arrived at the below formulae for deriving > > total machine capacity. > > > > Total Broker machine size = Message size per second * Retention period * > > Replication Factor > > > > Am I need to consider the topic, index files in the calculation? Please > > help/guide me if i missing any param required in the formulae. > > > > Index file calculation (Reference > > <https://issues.apache.org/jira/browse/KAFKA-3300>) > > > > Currently, the initial/max size of offset index file is configured by > > log.index.max.bytes. > > This will be the offset index file size for the active log segment until it > > rolls out. > > > > Theoretically, we can calculate the upper bound of offset index size using > > the following formula: > > > > log.segment.bytes / index.interval.bytes * 8 > > > > With the default setting the bytes needed for an offset index size is 1GB > > / 4K * 8 = 2MB. And the default log.index.max.bytes is 10MB. > > > > > > Retention Period = Retention period + (log.retention.check.interval.ms + > > log.segment.delete.delay.ms ) / 1000 > > = 86400 + (30000 + 6000)/1000 > > = 86400 + 360 > > = 86760 Seconds > > > > Total Broker machine size = Message size per second * Retention period * > > Replication Factor > > = 90 MB/Sec * 86760 * 3 > > = 23425200 MB > > = 23.4252 Tb > > > > *Machine Configuration* > > 6 Brokers with 3 ZK > > > > *Kafka (per machine)* > > Disk Space - 2 * 2TB > > RAM - 128 GB > > CPU - 40 core > > > > Thanks and Regards > > Gowtham.S > >
