Asking as I also need to do some of this math. How did you get to 90MB/s, the RAM size and CPU count ? You missing some dedicated SSD's for the ZK's Assume you doing licensing for 6 brokers... I'd suggest maybe rather go with 5 x 1TB drives, you provisioning 24TB for a 23.4 TB sizing, cutting it very close. What industry are you in, with 3 ZK's you can only handle one failure, assuming this infrastructure is going to be very critical, you already planning 6 machines, rather look at using 5 ZK's.
G > > Total Broker machine size = Message size per second * Retention period * > > Replication Factor > > = 90 MB/Sec * 86760 * 3 > > = 23425200 MB > > = 23.4252 Tb > > > > *Machine Configuration* > > 6 Brokers with 3 ZK > > > > *Kafka (per machine)* > > Disk Space - 2 * 2TB > > RAM - 128 GB > > CPU - 40 core On Tue, Feb 18, 2020 at 10:15 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hello Gowtham, > > You need to include the size of offset and time index files in your > calculations, plus potentially transaction indexes. > If you use default values, that means 10 MB each for every log > segment, which default size is itself 1 GB. > > Alexandre > > Le lun. 27 janv. 2020 à 08:51, Gowtham S <gowtham.co....@gmail.com> a > écrit : > > > > Hi, > > > > Please help/guide me to identify the size of the cluster. > > > > Thanks and Regards, > > Gowtham S. > > > > > > On Fri, 24 Jan 2020 at 14:31, Gowtham S <gowtham.co....@gmail.com> > wrote: > > > > > Hi, > > > > > > We are in the process of deploying Kafka in our service. We need to > decide > > > the machine capacity plan, we arrived at the below formulae for > deriving > > > total machine capacity. > > > > > > Total Broker machine size = Message size per second * Retention > period * > > > Replication Factor > > > > > > Am I need to consider the topic, index files in the calculation? Please > > > help/guide me if i missing any param required in the formulae. > > > > > > Index file calculation (Reference > > > <https://issues.apache.org/jira/browse/KAFKA-3300>) > > > > > > Currently, the initial/max size of offset index file is configured by > log.index.max.bytes. > > > This will be the offset index file size for the active log segment > until it > > > rolls out. > > > > > > Theoretically, we can calculate the upper bound of offset index size > using > > > the following formula: > > > > > > log.segment.bytes / index.interval.bytes * 8 > > > > > > With the default setting the bytes needed for an offset index size is > 1GB > > > / 4K * 8 = 2MB. And the default log.index.max.bytes is 10MB. > > > > > > > > > Retention Period = Retention period + (log.retention.check.interval.ms > + > > > log.segment.delete.delay.ms ) / 1000 > > > = 86400 + (30000 + 6000)/1000 > > > = 86400 + 360 > > > = 86760 Seconds > > > > > > Total Broker machine size = Message size per second * Retention > period * > > > Replication Factor > > > = 90 MB/Sec * 86760 * 3 > > > = 23425200 MB > > > = 23.4252 Tb > > > > > > *Machine Configuration* > > > 6 Brokers with 3 ZK > > > > > > *Kafka (per machine)* > > > Disk Space - 2 * 2TB > > > RAM - 128 GB > > > CPU - 40 core > > > > > > Thanks and Regards > > > Gowtham.S > > > > -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit!
