In Azure we recommend using managed disks for Kafka. HD Insight Kafka uses them. I generally see SSD for Kafka, but I guess part of that could depend on if you write larger writes from fewer publishers or small writes from many publishers. What does your workload look like?
Kind Regards, -Dan -----Original Message----- From: Ali Nazemian <alinazem...@gmail.com> Sent: Wednesday, July 4, 2018 6:58 AM To: users@kafka.apache.org Subject: Kafka disk recommendation for cloud Hi All, I was wondering what the recommendations are for disk type for hosting Kafka on a cloud environment? As far as I know, most of the best practices suggest using spinning disks for Kafka due to the fact that Kafka architecture relies on sequential write/read. Hence, the increase in Kafka performance by using SSD disks wouldn't be very cost-effective. However, on a cloud environment, it might be a different story due to hard limitations on IOPS. For an on-prem solution, the avg IOPS a spinning disk is very low (about 100-200), but when it comes to sequential IOPS can increase to 20k-30k based on different factors. However, for a cloud solution, there is a different story. For example, Azure limits every Spinning disk to 500 IOPS whether it is random or sequential. It means we should be able to get 500eps max per each disk, right? Therefore, does it mean using SSD disks for Kafka would be recommended for cloud providers? What about using RAID0 vs JBOD for Kafka Brokers? I can see various recommendations to use RAID0 or JBOD, but I am not really sure which one is recommended especially for a Cloud environment? Regards, Ali
