Thanks, George and Alexandre for the reply. George, to answer your question
> How did you get to 90MB/s - this is our expected throughput. For the RAM size and CPU count? > Can you please suggest me how to calculate this! With regards, Gowtham S, MCA PH: 9597000673 On Wed, 19 Feb 2020 at 09:27, George <george...@gmail.com> wrote: > Asking as I also need to do some of this math. > > How did you get to 90MB/s, the RAM size and CPU count ? > You missing some dedicated SSD's for the ZK's > Assume you doing licensing for 6 brokers... > I'd suggest maybe rather go with 5 x 1TB drives, you provisioning 24TB for > a 23.4 TB sizing, cutting it very close. > What industry are you in, with 3 ZK's you can only handle one failure, > assuming this infrastructure is going to be very critical, you already > planning 6 machines, rather look at using 5 ZK's. > > G > > > > Total Broker machine size = Message size per second * Retention > period * > > > Replication Factor > > > = 90 MB/Sec * 86760 * 3 > > > = 23425200 MB > > > = 23.4252 Tb > > > > > > *Machine Configuration* > > > 6 Brokers with 3 ZK > > > > > > *Kafka (per machine)* > > > Disk Space - 2 * 2TB > > > RAM - 128 GB > > > CPU - 40 core > > > On Tue, Feb 18, 2020 at 10:15 PM Alexandre Dupriez < > alexandre.dupr...@gmail.com> wrote: > > > Hello Gowtham, > > > > You need to include the size of offset and time index files in your > > calculations, plus potentially transaction indexes. > > If you use default values, that means 10 MB each for every log > > segment, which default size is itself 1 GB. > > > > Alexandre > > > > Le lun. 27 janv. 2020 à 08:51, Gowtham S <gowtham.co....@gmail.com> a > > écrit : > > > > > > Hi, > > > > > > Please help/guide me to identify the size of the cluster. > > > > > > Thanks and Regards, > > > Gowtham S. > > > > > > > > > On Fri, 24 Jan 2020 at 14:31, Gowtham S <gowtham.co....@gmail.com> > > wrote: > > > > > > > Hi, > > > > > > > > We are in the process of deploying Kafka in our service. We need to > > decide > > > > the machine capacity plan, we arrived at the below formulae for > > deriving > > > > total machine capacity. > > > > > > > > Total Broker machine size = Message size per second * Retention > > period * > > > > Replication Factor > > > > > > > > Am I need to consider the topic, index files in the calculation? > Please > > > > help/guide me if i missing any param required in the formulae. > > > > > > > > Index file calculation (Reference > > > > <https://issues.apache.org/jira/browse/KAFKA-3300>) > > > > > > > > Currently, the initial/max size of offset index file is configured by > > log.index.max.bytes. > > > > This will be the offset index file size for the active log segment > > until it > > > > rolls out. > > > > > > > > Theoretically, we can calculate the upper bound of offset index size > > using > > > > the following formula: > > > > > > > > log.segment.bytes / index.interval.bytes * 8 > > > > > > > > With the default setting the bytes needed for an offset index size is > > 1GB > > > > / 4K * 8 = 2MB. And the default log.index.max.bytes is 10MB. > > > > > > > > > > > > Retention Period = Retention period + ( > log.retention.check.interval.ms > > + > > > > log.segment.delete.delay.ms ) / 1000 > > > > = 86400 + (30000 + 6000)/1000 > > > > = 86400 + 360 > > > > = 86760 Seconds > > > > > > > > Total Broker machine size = Message size per second * Retention > > period * > > > > Replication Factor > > > > = 90 MB/Sec * 86760 * 3 > > > > = 23425200 MB > > > > = 23.4252 Tb > > > > > > > > *Machine Configuration* > > > > 6 Brokers with 3 ZK > > > > > > > > *Kafka (per machine)* > > > > Disk Space - 2 * 2TB > > > > RAM - 128 GB > > > > CPU - 40 core > > > > > > > > Thanks and Regards > > > > Gowtham.S > > > > > > > > > -- > You have the obligation to inform one honestly of the risk, and as a person > you are committed to educate yourself to the total risk in any activity! > > Once informed & totally aware of the risk, > every fool has the right to kill or injure themselves as they see fit! >
