We run many thousands of clusters on EC2 without notable issues, and achieve great performance there. The real thing that matters is how good your virtualization layer is and how much of a performance impact it has. E.g. in modern EC2, the performance overhead of using virtualized IO is around 1-5% tops, which isn't enough of an impact for kafka to really notice.
On Thu, Nov 30, 2017 at 11:56 AM, Wim Van Leuven < wim.vanleu...@highestpoint.biz> wrote: > We are running kafka on openstack for a testing/staging environment. > > It runs good and stable, but it obviously is way slower than bare-metal. > Simple reason is the distance to the disk (as with any IO batch oriented > system on virtualisation) and virtual network. > > HTH > -wim > > > On Thu, 30 Nov 2017 at 11:22 Viktor Somogyi <viktorsomo...@gmail.com> > wrote: > > > Hi folks, > > > > Recently I bumped into an interesting question: using kafka in > virtualized > > environments, such as vmware. I'm not really familiar with virtualization > > in-depth (how disk virtualization works, what are the OS level supports > > etc.), therefore I think this is an interesting discussion from Kafka's > > point. As far as I know Kafka is designed for a non-virtualized > environment > > mainly (although I haven't seen it explicitly anywhere) but thinking of > > it's hard reliance on disk optimization I always assumed this. > > > > Anyone has experiences with virtualized Kafka? Are you aware of any pain > > points that people should consider (or performance issues)? > > Are there any publications on this topic? > > > > Regards, > > Viktor > > >
