Hi, I have read in the note below
https://www.quora.com/What-is-viable-hardware-for-Zookeeper-and-Kafka-brokers Which states "Take your expected message size * expected messages/second, and multiply that by how many seconds you would like to keep your messages available in your kafka cluster for. Then multiply that size by two or more depending on what you choose your replication factor (ie: redundancy) to be, and divide that by how many brokers you'll have in your cluster. This will give you how much hard drive space per broker that you need. Kafka is a sequential message queue - there aren't any random reads and writes. This means that it can take heavy advantage of disk read-ahead, disk buffering, and linux kernel/filesystem optimizations. A typical hard disk doing sequential reads/writes is easily able to saturate a gigabit link, and so is Kafka with any decent CPU made in the last 5 years. This means that you can get by with big, cheap, slow disks. Please remember that the disk buffer cache lives in your RAM. That means that you need sufficient RAM in order to store a certain time period of messages in cache. If you're not doing anything crazy and are always expecting your consumers to be up to date, then you don't have to worry too much. Just keep this in mind should your consumers fall behind far enough that they need to retrieve data outside of buffer cache. I personally would use the money saved on disk to buy more RAM." This pretty old (May 2015) and I am wondering if it refers to Kafka < 0.8). However, the calculation's seems to be plausible. Any comments on it please? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
