I am currently running a few clusters on virtual machines connected to a VSAN. I have not had any major ssues. My clusters are small; 3 virtual machines (4 cpu and 8 GB memory each). I am routinely processing 20K msg/sec with surges of over 35K msg/sec. Based on my performance testing, I don't think my cluster would have issues until I start reaching about 80K msg/sec. Your message throughput will be highly dependent on your message size and whether you are compressing messages. You can test our the performance of your cluster using kafka-producer-perf-test and kafka-consumer-perf-test. This should give you a good idea of what your configuration can do.
The only issue I've had running on vms or a vsan is when VMWare itself has issues. Our admin had to reshuffle the VSAN one day and that caused high IOWait on my cluster and caused a number of partitions to become out of sync with the leader. On Sun, May 5, 2019 at 2:50 PM Jean-Marc Hyppolite < jean.marc.hyppol...@outlook.com> wrote: > Hello, > > I would like to know the impact of running kafka in production on virtual > machines connected to a SAN (Storage Area Network)? (I mean the impact on > kafka performance, what would be a max limit in terms of number of messages > per second, number of producers and consumers, kafka can deal with ...). If > we have no choice of running kafka on that kind of "hardware" what would be > the deployment guidelines along the with the max limit of number of > messages per second, of producers and consumers? > > Thank you. > > Jean-Marc. >
