Hi Liam, Thanks a lot for your response and it really helps for my assessment.
And also I am mentioning some additional information about my requirement, in case if you have additional information to provide kindly let me know. No of messages(including all topics messages) per day = 20000 Maximum size of each message = 1KB Total size of messages per day = 20000*1KB = 20MB Message retention = 5 days Number of topics = 4 (4 partitions are enough) And thinking to use below configuration. RAM to run Kafka cluster = 16GB (For all nodes including Kafka (2 nodes) and Zookeeper (2 nodes)) SD space= 20GB Thanks and regards, Naveen On Thu, Feb 20, 2020 at 9:37 AM Liam Clarke <liam.cla...@adscale.co.nz> wrote: > Hi Naveen, a very broad question, but to use Kafka as a backbone of your > infrastructure, the brokers need to be on machines with enough disk to > store the expected data, and with good network interface capacity - we use > 10Gbps NICs. > > Sizing disks is a case of knowing how long you want retain data for, how > many replicas you want to have, and how much data you expect, and how > you'll partition it. > > For example, if your topic will have 1TB of data produced onto it in a day, > and you want to have 3 replicas, retaining data for 14 days, you'll need 3 > * 14 * 1TB disk space across your cluster to fit it. > > But you also need to ensure that one partition can fit on disk on a broker. > So if the above topic has 20 partitions, you'll need to make sure that each > broker has a disk that can fit 50GB * 14. Plus you need some additional > space for Kafka metadata and OS overhead etc. > > Hope that helps, > > Liam Clarke > > On Thu, 20 Feb. 2020, 7:02 pm Naveen Kumar M, <eai.naveenkuma...@gmail.com > > > wrote: > > > Hello Kafka Community, > > > > Kindly help me on this. > > > > Thanks and regards, > > Naveen > > > > On Wed, Feb 19, 2020, 1:41 PM Naveen Kumar M < > eai.naveenkuma...@gmail.com> > > wrote: > > > > > Hello Team, > > > > > > Kindly help me to understand hardware assessment details to setup Kafka > > as > > > messaging broker to de-couple applications/systems. > > > > > > Thanks and regards, > > > Naveen > > > > > >
