That's correct, if you're mostly dealing with "latest" message consumption, faster disks will be mostly worthless. You will get some benefit if you have to rebalance partitions, since the cluster needs to shuffle a lot of data around for that, but during normal operations, there will be no benefit. Virtualization should have no impact on disk usage patterns/IOPS, if the VMs have low disk usage in general, but I don't have experience with that, so I'll leave it at that.
Those machines you describe are huge, 40 cores and 512GB of memory. You can probably run a whole RF3 cluster on just three machines :) However, you also don't mention any particular volume. How large are the messages you're sending? Compressed/uncompressed? Knowing who will be in charge of compression/decompression (producers/consumers or servers) is also important. Ballpark estimated number of messages going through the cluster per second at peak? You also don't mention bandwidth, and that's an important component. Remember that data has to be replicated throughout the cluster, so you should also take that into account. Having at least 10Gbe interfaces (we've done dual bonded 10GBe interfaces for 20Gb effective throughput) is usually a good recommendation for a production cluster. I've run into more problems with bandwidth than CPU/RAM issues, since Kafka can easily peg bandwidth usage with low CPU/RAM overhead. I still think the bare metal approach for Kafka servers is the best. I have yet to be convinced otherwise, and my limited experience with containers/virtualization of late has also reaffirmed this belief, at least for server-level type hardware. Marcos On Thu, Apr 13, 2017 at 1:02 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > Thank you very much, Marcos. > > My application is real-time processing so I would say most of the times I > am dealing with the "latest" message that emphasize page caching. In this > case, does it mean there is no additional throughput provided by using 10k > or 15k disks? What about having virtualized cluster for such a use case? > Obviously, virtualization does not affect page caching significantly, and > it seems I am dealing with the Memory rather than Disk most of the times. > The problem I've got here is it would be more consistent to have a single > hardware spec for everything and apparently 40 cores and 512GB memory would > be a lot for hosting Kafka or Storm separately. So I have to choose between > using bare metal and colocate Storm, Kafka, or use the virtualized at the > first place. What do you think, which one do you think would be better for > a real-time application with a hard SLA? > > Regards, > Ali > > On Thu, Apr 13, 2017 at 10:57 AM, Marcos Juarez <mjua...@gmail.com> wrote: > > > Ali, > > > > I don't know of proper benchmarks out there, but I've done some work in > > this area, when trying to determine what hardware to get for particular > use > > cases. My answers are in-line: > > > > On Mon, Apr 10, 2017 at 7:05 PM, Ali Nazemian <alinazem...@gmail.com> > > wrote: > > > > > Hi all, > > > > > > I was wondering if there is any benchmark or any recommendation for > > having > > > physical HW vs virtual for the Kafka Brokers. I am trying to calculate > > the > > > HW requirements for a Kafka Cluster with a hard SLA. My questions are > as > > > follows. > > > > > > - What is the effect of OS disk caching for a Kafka-Broker? How much > > > on-heap and off-heap memory would be required per node? > > > > > > > If by "OS disk caching" you mean page cache, then it's huge. Kafka > relies > > on it to serve as much data as possible from memory directly. The actual > > on-heap vs off-heap RAM requirements will be completely dependent on your > > scenario. I've run Kafka brokers with heaps as little as 8GB of RAM, and > > as much as 32GB of RAM each, using CMS GC with some customized tuning. > The > > boxes usually have much more physical RAM than that (64GB of RAM and > 128GB > > respectively), for page cache. > > > > I think the Confluent recommendation these days is to use G1 GC, but I've > > got no experience using that. > > > > > > > > > > - Since Kafka read-write workload is pretty sequential which of the > > > following spinning disks would be recommended? SATA 7.2k, SAS 10k, SAS > > 15k? > > > > > > > Again, that will depend on your use case. Are your consumers mostly > > consuming up-to-the-second messages, or are they always connecting and > > consuming from arbitrary offsets, or even from the beginning of topics? > If > > your consumers are always consuming latest, then actual disk IO will be > > extremely low, since almost everything will be served out of page > > cache/memory, and most of the activity will be fsync's every ~5 seconds > or > > so. On the other hand, if most of your consumption will be of old data, > > then the brokers will need to read a large amount of data from disk > > sequentially whenever consumers request data. That's when you'd benefit > > from something like SAS 10k or SAS 15k. You'd need to run some custom > > benchmarks to figure out what would work on workload that mimics your use > > case. > > > > > > > > > > - Since Kafka is not CPU-intensive, how bad would be to coexist > > > Kafka-Broker and a CPU-intensive workload like STORM? > > > > > > > I wouldn't recommend that at all. I've always run Kafka on bare metal > with > > nothing else running on that box. Otherwise, you won't be able to > identify > > bottlenecks when you run into them, and both systems would be impacting > > each other all the time, potentially. Hardware is cheap, not worth it to > > spend time chasing issues caused by "noisy neighbors" on the boxes. > > > > > > > > > > Regards, > > > Ali > > > > > > > Marcos > > > > > > -- > A.Nazemian >
