Hi Liam, Thanks a lot for valuable information and this really helps for my assessment.
Thanks and Regards, Naveen On Thu, Feb 20, 2020, 8:38 PM Liam Clarke <liam.cla...@adscale.co.nz> wrote: > The specs of your broker machines look fine for your use case. But you'll > need to run 3 > ZK nodes at least so that ZK can maintain quorum in the event of a node > failure, network partition etc. that removes a node. With two ZK nodes, one > failing node would take out your Zookeeper cluster, as quorum requires a > majority of nodes to be available. > > https://mapr.com/docs/61/MapROverview/c_zookeeper.html > > On Thu, 20 Feb. 2020, 11:51 pm Naveen Kumar M, < > eai.naveenkuma...@gmail.com> > wrote: > > > Hi Liam, > > > > Thanks a lot for your response and it really helps for my assessment. > > > > And also I am mentioning some additional information about my > requirement, > > in case if you have additional information to provide kindly let me know. > > > > No of messages(including all topics messages) per day = 20000 > > Maximum size of each message = 1KB > > Total size of messages per day = 20000*1KB = 20MB > > Message retention = 5 days > > Number of topics = 4 (4 partitions are enough) > > > > And thinking to use below configuration. > > > > RAM to run Kafka cluster = 16GB (For all nodes including Kafka (2 nodes) > > and Zookeeper (2 nodes)) > > SD space= 20GB > > > > Thanks and regards, > > Naveen > > > > > > On Thu, Feb 20, 2020 at 9:37 AM Liam Clarke <liam.cla...@adscale.co.nz> > > wrote: > > > > > Hi Naveen, a very broad question, but to use Kafka as a backbone of > your > > > infrastructure, the brokers need to be on machines with enough disk to > > > store the expected data, and with good network interface capacity - we > > use > > > 10Gbps NICs. > > > > > > Sizing disks is a case of knowing how long you want retain data for, > how > > > many replicas you want to have, and how much data you expect, and how > > > you'll partition it. > > > > > > For example, if your topic will have 1TB of data produced onto it in a > > day, > > > and you want to have 3 replicas, retaining data for 14 days, you'll > > need 3 > > > * 14 * 1TB disk space across your cluster to fit it. > > > > > > But you also need to ensure that one partition can fit on disk on a > > broker. > > > So if the above topic has 20 partitions, you'll need to make sure that > > each > > > broker has a disk that can fit 50GB * 14. Plus you need some additional > > > space for Kafka metadata and OS overhead etc. > > > > > > Hope that helps, > > > > > > Liam Clarke > > > > > > On Thu, 20 Feb. 2020, 7:02 pm Naveen Kumar M, < > > eai.naveenkuma...@gmail.com > > > > > > > wrote: > > > > > > > Hello Kafka Community, > > > > > > > > Kindly help me on this. > > > > > > > > Thanks and regards, > > > > Naveen > > > > > > > > On Wed, Feb 19, 2020, 1:41 PM Naveen Kumar M < > > > eai.naveenkuma...@gmail.com> > > > > wrote: > > > > > > > > > Hello Team, > > > > > > > > > > Kindly help me to understand hardware assessment details to setup > > Kafka > > > > as > > > > > messaging broker to de-couple applications/systems. > > > > > > > > > > Thanks and regards, > > > > > Naveen > > > > > > > > > > > > > > >
