The physical memory you need depends on the type of workload you are running and the particular setup for this workload, e.g. number of partitions, and production and consumption patterns.
Assuming you have 90 MB/s of *production* throughput, a replication factor of 3 across topics, even repartition of and load on partitions across brokers, that put 270 MB/s of steady disk throughput. If it happens you reach or exceed the sustainable disk throughput you will need to have enough memory to hold the temporary excess of data in the page cache so as to not impact producers negatively (i.e. apparition of URPS, lowered produce throughput and higher latency variance for producers) which can happen when you don't have enough free memory available and record append to segments start blocking on disk operations. Now, consumers need to be taken into account as well. Optimal performance for serving consumers can be achieved if data is still in the page cache. And this depends on how far in the past consumers are fetching data. The more precise you would like to get in your capacity planning, the more you would need performance testing with production-like workload to fine-tune the required hardware and configuration of your cluster. Le mer. 19 févr. 2020 à 05:31, George <george...@gmail.com> a écrit : > > Hi there > > with regard to "> Can you please suggest me how to calculate this!" > > No I can't thats why I asked, getting idea only way you going to figure > this out is by building 6 servers, with say 12-16 vCPU's assigned and then > run test/volume load and see what Prometheus shows you from a CPU/memory > Utilisation. > > The 90Mb/s, how did you calculate that? > > G > > > > On Wed, Feb 19, 2020 at 7:29 AM Gowtham S <gowtham.co....@gmail.com> wrote: > > > Thanks, George and Alexandre for the reply. > > > > George, to answer your question > > > > > How did you get to 90MB/s - this is our expected throughput. > > > > > > For the RAM size and CPU count? > > > > > Can you please suggest me how to calculate this! > > > > > > > > With regards, > > Gowtham S, MCA > > PH: 9597000673 > > > > > > On Wed, 19 Feb 2020 at 09:27, George <george...@gmail.com> wrote: > > > > > Asking as I also need to do some of this math. > > > > > > How did you get to 90MB/s, the RAM size and CPU count ? > > > You missing some dedicated SSD's for the ZK's > > > Assume you doing licensing for 6 brokers... > > > I'd suggest maybe rather go with 5 x 1TB drives, you provisioning 24TB > > for > > > a 23.4 TB sizing, cutting it very close. > > > What industry are you in, with 3 ZK's you can only handle one failure, > > > assuming this infrastructure is going to be very critical, you already > > > planning 6 machines, rather look at using 5 ZK's. > > > > > > G > > > > > > > > Total Broker machine size = Message size per second * Retention > > > period * > > > > > Replication Factor > > > > > = 90 MB/Sec * 86760 * 3 > > > > > = 23425200 MB > > > > > = 23.4252 Tb > > > > > > > > > > *Machine Configuration* > > > > > 6 Brokers with 3 ZK > > > > > > > > > > *Kafka (per machine)* > > > > > Disk Space - 2 * 2TB > > > > > RAM - 128 GB > > > > > CPU - 40 core > > > > > > > > > On Tue, Feb 18, 2020 at 10:15 PM Alexandre Dupriez < > > > alexandre.dupr...@gmail.com> wrote: > > > > > > > Hello Gowtham, > > > > > > > > You need to include the size of offset and time index files in your > > > > calculations, plus potentially transaction indexes. > > > > If you use default values, that means 10 MB each for every log > > > > segment, which default size is itself 1 GB. > > > > > > > > Alexandre > > > > > > > > Le lun. 27 janv. 2020 à 08:51, Gowtham S <gowtham.co....@gmail.com> a > > > > écrit : > > > > > > > > > > Hi, > > > > > > > > > > Please help/guide me to identify the size of the cluster. > > > > > > > > > > Thanks and Regards, > > > > > Gowtham S. > > > > > > > > > > > > > > > On Fri, 24 Jan 2020 at 14:31, Gowtham S <gowtham.co....@gmail.com> > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > We are in the process of deploying Kafka in our service. We need to > > > > decide > > > > > > the machine capacity plan, we arrived at the below formulae for > > > > deriving > > > > > > total machine capacity. > > > > > > > > > > > > Total Broker machine size = Message size per second * Retention > > > > period * > > > > > > Replication Factor > > > > > > > > > > > > Am I need to consider the topic, index files in the calculation? > > > Please > > > > > > help/guide me if i missing any param required in the formulae. > > > > > > > > > > > > Index file calculation (Reference > > > > > > <https://issues.apache.org/jira/browse/KAFKA-3300>) > > > > > > > > > > > > Currently, the initial/max size of offset index file is configured > > by > > > > log.index.max.bytes. > > > > > > This will be the offset index file size for the active log segment > > > > until it > > > > > > rolls out. > > > > > > > > > > > > Theoretically, we can calculate the upper bound of offset index > > size > > > > using > > > > > > the following formula: > > > > > > > > > > > > log.segment.bytes / index.interval.bytes * 8 > > > > > > > > > > > > With the default setting the bytes needed for an offset index size > > is > > > > 1GB > > > > > > / 4K * 8 = 2MB. And the default log.index.max.bytes is 10MB. > > > > > > > > > > > > > > > > > > Retention Period = Retention period + ( > > > log.retention.check.interval.ms > > > > + > > > > > > log.segment.delete.delay.ms ) / 1000 > > > > > > = 86400 + (30000 + 6000)/1000 > > > > > > = 86400 + 360 > > > > > > = 86760 Seconds > > > > > > > > > > > > Total Broker machine size = Message size per second * Retention > > > > period * > > > > > > Replication Factor > > > > > > = 90 MB/Sec * 86760 * 3 > > > > > > = 23425200 MB > > > > > > = 23.4252 Tb > > > > > > > > > > > > *Machine Configuration* > > > > > > 6 Brokers with 3 ZK > > > > > > > > > > > > *Kafka (per machine)* > > > > > > Disk Space - 2 * 2TB > > > > > > RAM - 128 GB > > > > > > CPU - 40 core > > > > > > > > > > > > Thanks and Regards > > > > > > Gowtham.S > > > > > > > > > > > > > > > > > > > -- > > > You have the obligation to inform one honestly of the risk, and as a > > person > > > you are committed to educate yourself to the total risk in any activity! > > > > > > Once informed & totally aware of the risk, > > > every fool has the right to kill or injure themselves as they see fit! > > > > > > > > -- > You have the obligation to inform one honestly of the risk, and as a person > you are committed to educate yourself to the total risk in any activity! > > Once informed & totally aware of the risk, > every fool has the right to kill or injure themselves as they see fit!
