The specs of your broker machines look fine for your use case. But you'll need to run 3 ZK nodes at least so that ZK can maintain quorum in the event of a node failure, network partition etc. that removes a node. With two ZK nodes, one failing node would take out your Zookeeper cluster, as quorum requires a majority of nodes to be available.
https://mapr.com/docs/61/MapROverview/c_zookeeper.html On Thu, 20 Feb. 2020, 11:51 pm Naveen Kumar M, <eai.naveenkuma...@gmail.com> wrote: > Hi Liam, > > Thanks a lot for your response and it really helps for my assessment. > > And also I am mentioning some additional information about my requirement, > in case if you have additional information to provide kindly let me know. > > No of messages(including all topics messages) per day = 20000 > Maximum size of each message = 1KB > Total size of messages per day = 20000*1KB = 20MB > Message retention = 5 days > Number of topics = 4 (4 partitions are enough) > > And thinking to use below configuration. > > RAM to run Kafka cluster = 16GB (For all nodes including Kafka (2 nodes) > and Zookeeper (2 nodes)) > SD space= 20GB > > Thanks and regards, > Naveen > > > On Thu, Feb 20, 2020 at 9:37 AM Liam Clarke <liam.cla...@adscale.co.nz> > wrote: > > > Hi Naveen, a very broad question, but to use Kafka as a backbone of your > > infrastructure, the brokers need to be on machines with enough disk to > > store the expected data, and with good network interface capacity - we > use > > 10Gbps NICs. > > > > Sizing disks is a case of knowing how long you want retain data for, how > > many replicas you want to have, and how much data you expect, and how > > you'll partition it. > > > > For example, if your topic will have 1TB of data produced onto it in a > day, > > and you want to have 3 replicas, retaining data for 14 days, you'll > need 3 > > * 14 * 1TB disk space across your cluster to fit it. > > > > But you also need to ensure that one partition can fit on disk on a > broker. > > So if the above topic has 20 partitions, you'll need to make sure that > each > > broker has a disk that can fit 50GB * 14. Plus you need some additional > > space for Kafka metadata and OS overhead etc. > > > > Hope that helps, > > > > Liam Clarke > > > > On Thu, 20 Feb. 2020, 7:02 pm Naveen Kumar M, < > eai.naveenkuma...@gmail.com > > > > > wrote: > > > > > Hello Kafka Community, > > > > > > Kindly help me on this. > > > > > > Thanks and regards, > > > Naveen > > > > > > On Wed, Feb 19, 2020, 1:41 PM Naveen Kumar M < > > eai.naveenkuma...@gmail.com> > > > wrote: > > > > > > > Hello Team, > > > > > > > > Kindly help me to understand hardware assessment details to setup > Kafka > > > as > > > > messaging broker to de-couple applications/systems. > > > > > > > > Thanks and regards, > > > > Naveen > > > > > > > > > >
